feat: Update to Pytorch 2.0 #1528
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Dheeraj Peri <[email protected]>
github-actions
bot
added
component: api [Python]
Signed-off-by: Dheeraj Peri <[email protected]>
Signed-off-by: Dheeraj Peri <[email protected]>
|
@frank-wei We are observing FX test failures with latest pytorch upgrade. Any insights into this ? Thanks !! https://github.com/pytorch/TensorRT/blob/master/py/torch_tensorrt/fx/test/converters/aten_op/test_flatten_aten.py#L30 needs to be changed ? |
These are PT2.0 tracer based tests. The reason is that PT2.0 changed the aten._reshape_alias to aten.view. I will fix those tests. |
7 tasks
|
Should be fixed in #1559 |
github-actions
bot
added
component: conversion
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2022-12-17 20:08:18.017950 +0000
+++ tools/perf/perf_run.py 2022-12-17 20:08:36.883639 +0000
@@ -614,14 +614,22 @@
+ "(among the following options vgg16|resnet50|efficientnet_b0|vit) "
+ "or provide a torch model file"
)
if use_dynamo and (model_torch is None):
- raise ValueError("No Pytorch model (nn.Module) is provided for torchdynamo compilation. Please provide a pytorch model")
+ raise ValueError(
+ "No Pytorch model (nn.Module) is provided for torchdynamo compilation. Please provide a pytorch model"
+ )
if use_dynamo and model_torch:
- model_torch = torch.compile(model_torch, "default", dynamic=False, fullgraph=False, backend=dynamo_backend)
+ model_torch = torch.compile(
+ model_torch,
+ "default",
+ dynamic=False,
+ fullgraph=False,
+ backend=dynamo_backend,
+ )
backends = parse_backends(params["backends"])
truncate_long_and_double = params["truncate"]
batch_size = params["batch_size"]
is_trt_engine = params["is_trt_engine"]Signed-off-by: Dheeraj Peri <[email protected]>
- Fix compilation error for GPT-2 model arising from Byte-type inputs fed into TensorRT Engine - Update translation dictionary between Torch and TensorRT types to include `at::kByte` - Add field to PartitioningInfo specifying whether to cast Int8 inputs to TensorRT Engines to Int, to avoid error arising from Int8 inputs being fed into non-quantized engines - Add automatic detection of quantized/calibrated models and disable Int8 => Int32 casting in those cases - Fix bug where LoweringInfo target device was not being updated for Python API - Allow `castNode` to force creation of a new node and avoid searching for an existing one to convert - Add test to ensure cast is inserted in the Torch engine preceding a TensorRT engine, when the Byte tensor is an output of the Torch engine
Signed-off-by: Dheeraj Peri <[email protected]>
- Address review comments - Improve documentation and logging messages - Restructure casting function to allow for casting of variable data types - Add casting for `at::kByte` segment block inputs as well as segment block outputs
fix: Properly cast intermediate Int8 tensors to TensorRT Engines in Fallback
Signed-off-by: Torch-TensorRT Github Bot <[email protected]>
Signed-off-by: Dheeraj Peri <[email protected]>
github-actions
bot
added
component: api [C++]
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2022-12-28 21:02:32.955162 +0000
+++ tools/perf/perf_run.py 2022-12-28 21:02:49.724464 +0000
@@ -15,10 +15,11 @@
import pandas as pd
# Importing supported Backends
import torch
import torch_tensorrt as torchtrt
+
# from torch_tensorrt.fx.lower import compile
# from torch_tensorrt.fx.utils import LowerPrecision
import tensorrt as trt
from utils import (
@@ -142,11 +143,11 @@
start_compile = time.time_ns()
model = torchtrt.compile(
model,
ir="fx",
inputs=input_tensors,
- enabled_precisions={torch.float16 if precision=="fp16" else torch.float32},
+ enabled_precisions={torch.float16 if precision == "fp16" else torch.float32},
)
end_compile = time.time_ns()
compile_time_ms = (end_compile - start_compile) / 1e6
iters = params.get("iterations", 20)
@@ -167,16 +168,24 @@
meas_time = end_time - start_time
timings.append(meas_time)
recordStats("FX-TensorRT", timings, precision, batch_size, compile_time_ms)
+
def run_dynamo(model, input_tensors, params, precision, batch_size):
dynamo_backend = params["dynamo_backend"]
- print("Running Dynamo with backend: ", dynamo_backend, " for precision: ", precision, " batch_size : ", batch_size)
+ print(
+ "Running Dynamo with backend: ",
+ dynamo_backend,
+ " for precision: ",
+ precision,
+ " batch_size : ",
+ batch_size,
+ )
if precision == "fp16":
- input_tensors = [tensor.half() for tensor in input_tensors]
+ input_tensors = [tensor.half() for tensor in input_tensors]
fp16_mode = True if precision == "fp16" else False
# dynamo_backend_params = {"fp16_mode" : fp16_mode}
# model = torch.compile(
# model,
@@ -185,10 +194,11 @@
# fullgraph=False,
# backend=dynamo_backend,
# # **dynamo_backend_params
# )
import torch._dynamo as dynamo
+
model = dynamo.optimize(dynamo_backend, nopython=True)(model)
# Compile and measure the time
with torch.no_grad():
start_compile = time.time_ns()
features = model(*input_tensors)
@@ -217,11 +227,14 @@
torch.cuda.synchronize()
end_time = timeit.default_timer()
meas_time = end_time - start_time
timings.append(meas_time)
- recordStats("Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms)
+ recordStats(
+ "Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms
+ )
+
def torch_dtype_from_trt(dtype):
if dtype == trt.int8:
return torch.int8
elif dtype == trt.bool:Signed-off-by: Dheeraj Peri <[email protected]>
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2022-12-28 21:05:04.918867 +0000
+++ tools/perf/perf_run.py 2022-12-28 21:05:22.129868 +0000
@@ -15,10 +15,11 @@
import pandas as pd
# Importing supported Backends
import torch
import torch_tensorrt as torchtrt
+
# from torch_tensorrt.fx.lower import compile
# from torch_tensorrt.fx.utils import LowerPrecision
import tensorrt as trt
from utils import (
@@ -142,11 +143,11 @@
start_compile = time.time_ns()
model = torchtrt.compile(
model,
ir="fx",
inputs=input_tensors,
- enabled_precisions={torch.float16 if precision=="fp16" else torch.float32},
+ enabled_precisions={torch.float16 if precision == "fp16" else torch.float32},
)
end_compile = time.time_ns()
compile_time_ms = (end_compile - start_compile) / 1e6
iters = params.get("iterations", 20)
@@ -167,16 +168,24 @@
meas_time = end_time - start_time
timings.append(meas_time)
recordStats("FX-TensorRT", timings, precision, batch_size, compile_time_ms)
+
def run_dynamo(model, input_tensors, params, precision, batch_size):
dynamo_backend = params["dynamo_backend"]
- print("Running Dynamo with backend: ", dynamo_backend, " for precision: ", precision, " batch_size : ", batch_size)
+ print(
+ "Running Dynamo with backend: ",
+ dynamo_backend,
+ " for precision: ",
+ precision,
+ " batch_size : ",
+ batch_size,
+ )
if precision == "fp16":
- input_tensors = [tensor.half() for tensor in input_tensors]
+ input_tensors = [tensor.half() for tensor in input_tensors]
fp16_mode = True if precision == "fp16" else False
# dynamo_backend_params = {"fp16_mode" : fp16_mode}
# model = torch.compile(
# model,
@@ -185,10 +194,11 @@
# fullgraph=False,
# backend=dynamo_backend,
# # **dynamo_backend_params
# )
import torch._dynamo as dynamo
+
model = dynamo.optimize(dynamo_backend, nopython=True)(model)
# Compile and measure the time
with torch.no_grad():
start_compile = time.time_ns()
features = model(*input_tensors)
@@ -217,11 +227,14 @@
torch.cuda.synchronize()
end_time = timeit.default_timer()
meas_time = end_time - start_time
timings.append(meas_time)
- recordStats("Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms)
+ recordStats(
+ "Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms
+ )
+
def torch_dtype_from_trt(dtype):
if dtype == trt.int8:
return torch.int8
elif dtype == trt.bool:Signed-off-by: Dheeraj Peri <[email protected]>
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2022-12-30 03:03:32.906616 +0000
+++ tools/perf/perf_run.py 2022-12-30 03:03:50.481014 +0000
@@ -15,10 +15,11 @@
import pandas as pd
# Importing supported Backends
import torch
import torch_tensorrt as torchtrt
+
# from torch_tensorrt.fx.lower import compile
# from torch_tensorrt.fx.utils import LowerPrecision
import tensorrt as trt
from utils import (
@@ -142,11 +143,11 @@
start_compile = time.time_ns()
model = torchtrt.compile(
model,
ir="fx",
inputs=input_tensors,
- enabled_precisions={torch.float16 if precision=="fp16" else torch.float32},
+ enabled_precisions={torch.float16 if precision == "fp16" else torch.float32},
)
end_compile = time.time_ns()
compile_time_ms = (end_compile - start_compile) / 1e6
iters = params.get("iterations", 20)
@@ -167,16 +168,24 @@
meas_time = end_time - start_time
timings.append(meas_time)
recordStats("FX-TensorRT", timings, precision, batch_size, compile_time_ms)
+
def run_dynamo(model, input_tensors, params, precision, batch_size):
dynamo_backend = params["dynamo_backend"]
- print("Running Dynamo with backend: ", dynamo_backend, " for precision: ", precision, " batch_size : ", batch_size)
+ print(
+ "Running Dynamo with backend: ",
+ dynamo_backend,
+ " for precision: ",
+ precision,
+ " batch_size : ",
+ batch_size,
+ )
if precision == "fp16":
- input_tensors = [tensor.half() for tensor in input_tensors]
+ input_tensors = [tensor.half() for tensor in input_tensors]
fp16_mode = True if precision == "fp16" else False
# dynamo_backend_params = {"fp16_mode" : fp16_mode}
# model = torch.compile(
# model,
@@ -185,10 +194,11 @@
# fullgraph=False,
# backend=dynamo_backend,
# # **dynamo_backend_params
# )
import torch._dynamo as dynamo
+
model = dynamo.optimize(dynamo_backend, nopython=True)(model)
# Compile and measure the time
with torch.no_grad():
start_compile = time.time_ns()
features = model(*input_tensors)
@@ -217,11 +227,14 @@
torch.cuda.synchronize()
end_time = timeit.default_timer()
meas_time = end_time - start_time
timings.append(meas_time)
- recordStats("Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms)
+ recordStats(
+ "Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms
+ )
+
def torch_dtype_from_trt(dtype):
if dtype == trt.int8:
return torch.int8
elif dtype == trt.bool:Signed-off-by: Dheeraj Peri <[email protected]>
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2023-01-03 20:57:19.467070 +0000
+++ tools/perf/perf_run.py 2023-01-03 20:57:40.994896 +0000
@@ -15,10 +15,11 @@
import pandas as pd
# Importing supported Backends
import torch
import torch_tensorrt as torchtrt
+
# from torch_tensorrt.fx.lower import compile
# from torch_tensorrt.fx.utils import LowerPrecision
import tensorrt as trt
from utils import (
@@ -142,11 +143,11 @@
start_compile = time.time_ns()
model = torchtrt.compile(
model,
ir="fx",
inputs=input_tensors,
- enabled_precisions={torch.float16 if precision=="fp16" else torch.float32},
+ enabled_precisions={torch.float16 if precision == "fp16" else torch.float32},
)
end_compile = time.time_ns()
compile_time_ms = (end_compile - start_compile) / 1e6
iters = params.get("iterations", 20)
@@ -167,16 +168,24 @@
meas_time = end_time - start_time
timings.append(meas_time)
recordStats("FX-TensorRT", timings, precision, batch_size, compile_time_ms)
+
def run_dynamo(model, input_tensors, params, precision, batch_size):
dynamo_backend = params["dynamo_backend"]
- print("Running Dynamo with backend: ", dynamo_backend, " for precision: ", precision, " batch_size : ", batch_size)
+ print(
+ "Running Dynamo with backend: ",
+ dynamo_backend,
+ " for precision: ",
+ precision,
+ " batch_size : ",
+ batch_size,
+ )
if precision == "fp16":
- input_tensors = [tensor.half() for tensor in input_tensors]
+ input_tensors = [tensor.half() for tensor in input_tensors]
fp16_mode = True if precision == "fp16" else False
# dynamo_backend_params = {"fp16_mode" : fp16_mode}
# model = torch.compile(
# model,
@@ -185,10 +194,11 @@
# fullgraph=False,
# backend=dynamo_backend,
# # **dynamo_backend_params
# )
import torch._dynamo as dynamo
+
model = dynamo.optimize(dynamo_backend, nopython=True)(model)
# Compile and measure the time
with torch.no_grad():
start_compile = time.time_ns()
features = model(*input_tensors)
@@ -217,11 +227,14 @@
torch.cuda.synchronize()
end_time = timeit.default_timer()
meas_time = end_time - start_time
timings.append(meas_time)
- recordStats("Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms)
+ recordStats(
+ "Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms
+ )
+
def torch_dtype_from_trt(dtype):
if dtype == trt.int8:
return torch.int8
elif dtype == trt.bool:Signed-off-by: Dheeraj Peri <[email protected]>
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2023-01-03 21:46:18.769567 +0000
+++ tools/perf/perf_run.py 2023-01-03 21:46:36.026528 +0000
@@ -15,10 +15,11 @@
import pandas as pd
# Importing supported Backends
import torch
import torch_tensorrt as torchtrt
+
# from torch_tensorrt.fx.lower import compile
# from torch_tensorrt.fx.utils import LowerPrecision
import tensorrt as trt
from utils import (
@@ -142,11 +143,11 @@
start_compile = time.time_ns()
model = torchtrt.compile(
model,
ir="fx",
inputs=input_tensors,
- enabled_precisions={torch.float16 if precision=="fp16" else torch.float32},
+ enabled_precisions={torch.float16 if precision == "fp16" else torch.float32},
)
end_compile = time.time_ns()
compile_time_ms = (end_compile - start_compile) / 1e6
iters = params.get("iterations", 20)
@@ -167,16 +168,24 @@
meas_time = end_time - start_time
timings.append(meas_time)
recordStats("FX-TensorRT", timings, precision, batch_size, compile_time_ms)
+
def run_dynamo(model, input_tensors, params, precision, batch_size):
dynamo_backend = params["dynamo_backend"]
- print("Running Dynamo with backend: ", dynamo_backend, " for precision: ", precision, " batch_size : ", batch_size)
+ print(
+ "Running Dynamo with backend: ",
+ dynamo_backend,
+ " for precision: ",
+ precision,
+ " batch_size : ",
+ batch_size,
+ )
if precision == "fp16":
- input_tensors = [tensor.half() for tensor in input_tensors]
+ input_tensors = [tensor.half() for tensor in input_tensors]
fp16_mode = True if precision == "fp16" else False
# dynamo_backend_params = {"fp16_mode" : fp16_mode}
# model = torch.compile(
# model,
@@ -185,10 +194,11 @@
# fullgraph=False,
# backend=dynamo_backend,
# # **dynamo_backend_params
# )
import torch._dynamo as dynamo
+
model = dynamo.optimize(dynamo_backend, nopython=True)(model)
# Compile and measure the time
with torch.no_grad():
start_compile = time.time_ns()
features = model(*input_tensors)
@@ -217,11 +227,14 @@
torch.cuda.synchronize()
end_time = timeit.default_timer()
meas_time = end_time - start_time
timings.append(meas_time)
- recordStats("Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms)
+ recordStats(
+ "Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms
+ )
+
def torch_dtype_from_trt(dtype):
if dtype == trt.int8:
return torch.int8
elif dtype == trt.bool:Signed-off-by: Dheeraj Peri <[email protected]>
There was a problem hiding this comment.
There are some changes that do not conform to Python style guidelines:
--- tools/perf/perf_run.py 2023-01-03 21:47:55.345180 +0000
+++ tools/perf/perf_run.py 2023-01-03 21:48:18.667455 +0000
@@ -15,10 +15,11 @@
import pandas as pd
# Importing supported Backends
import torch
import torch_tensorrt as torchtrt
+
# from torch_tensorrt.fx.lower import compile
# from torch_tensorrt.fx.utils import LowerPrecision
import tensorrt as trt
from utils import (
@@ -142,11 +143,11 @@
start_compile = time.time_ns()
model = torchtrt.compile(
model,
ir="fx",
inputs=input_tensors,
- enabled_precisions={torch.float16 if precision=="fp16" else torch.float32},
+ enabled_precisions={torch.float16 if precision == "fp16" else torch.float32},
)
end_compile = time.time_ns()
compile_time_ms = (end_compile - start_compile) / 1e6
iters = params.get("iterations", 20)
@@ -167,16 +168,24 @@
meas_time = end_time - start_time
timings.append(meas_time)
recordStats("FX-TensorRT", timings, precision, batch_size, compile_time_ms)
+
def run_dynamo(model, input_tensors, params, precision, batch_size):
dynamo_backend = params["dynamo_backend"]
- print("Running Dynamo with backend: ", dynamo_backend, " for precision: ", precision, " batch_size : ", batch_size)
+ print(
+ "Running Dynamo with backend: ",
+ dynamo_backend,
+ " for precision: ",
+ precision,
+ " batch_size : ",
+ batch_size,
+ )
if precision == "fp16":
- input_tensors = [tensor.half() for tensor in input_tensors]
+ input_tensors = [tensor.half() for tensor in input_tensors]
fp16_mode = True if precision == "fp16" else False
# dynamo_backend_params = {"fp16_mode" : fp16_mode}
# model = torch.compile(
# model,
@@ -185,10 +194,11 @@
# fullgraph=False,
# backend=dynamo_backend,
# # **dynamo_backend_params
# )
import torch._dynamo as dynamo
+
model = dynamo.optimize(dynamo_backend, nopython=True)(model)
# Compile and measure the time
with torch.no_grad():
start_compile = time.time_ns()
features = model(*input_tensors)
@@ -217,11 +227,14 @@
torch.cuda.synchronize()
end_time = timeit.default_timer()
meas_time = end_time - start_time
timings.append(meas_time)
- recordStats("Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms)
+ recordStats(
+ "Dynamo-" + dynamo_backend, timings, precision, batch_size, compile_time_ms
+ )
+
def torch_dtype_from_trt(dtype):
if dtype == trt.int8:
return torch.int8
elif dtype == trt.bool:Signed-off-by: Dheeraj Peri <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Dheeraj Peri [email protected]
Description
Upgrade to Pyt 2.0 (which is 1.14)
Fixes # (issue)
Type of change
Please delete options that are not relevant and/or add your own.
Checklist: