Is it possible to relax the dependency on `typing-extension < 4.6.0` #1753

andresliszt · 2023-09-27T16:38:06Z

I'm using some libraries such as pydantic that have conflicts with tensorflow-probability due to typing-extension dependency. For example pydantic depends on typing-extension >= 4.6.1. Same problem in tensorflow was solved with the new release 2.14

The text was updated successfully, but these errors were encountered:

csuter · 2023-09-27T18:14:28Z

thanks for raising -- testing removing this now; assuming it passes, we'll merge the change shortly.

csuter · 2023-09-27T18:37:52Z

hm this was not a trivial change -- it's causing some tests to fail. i don't have time to dig in deeply at the moment.

alexander-held · 2023-10-09T16:11:52Z

As a +1 here: I run into the same conflict with the upper bound when using typeguard (which introduces an incompatible lower bound).

SiegeLordEx · 2023-10-09T22:33:16Z

Here's the test failure that is blocking this, in case someone wishes to take a look. See also tensorflow/tensorflow#60687 and tensorflow/tensorflow#61387 which ostensibly solved it.

Reproduction steps:

Patch

diff --git a/required_packages.py b/required_packages.py
index cefd122..7652866 100644
--- a/required_packages.py
+++ b/required_packages.py
@@ -26,7 +26,7 @@ REQUIRED_PACKAGES = [
     'cloudpickle>=1.3',
     'gast>=0.3.2',  # For autobatching
     'dm-tree',  # For NumPy/JAX backends (hence, also for prefer_static)
-    'typing-extensions<4.6.0',  # TODO(b/284106340): Remove this pin
+    'typing-extensions',
 ]
 
 if __name__ == '__main__':

In a virtual env, run:

./testing/install_test_dependencies.sh
./testing/run_tfp_test.sh //tensorflow_probability/python/experimental/linalg:linear_operator_psd_kernel_test

With typing_extensions==4.8.0 this fails with:

ERROR: test_matmul_grad_xla_kernelparams_eager (__main__.LinearOperatorPSDKernelTest)                                                                                                                               
LinearOperatorPSDKernelTest.test_matmul_grad_xla_kernelparams_eager                                       
test_matmul_grad_xla_kernelparams_eager(mode='eager', tf_function='')                                     
----------------------------------------------------------------------                                                                                                                                              
Traceback (most recent call last):
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/absl/testing/parameterized.py", line 319, in bound_param_test                                                                                      
    return test_method(self, **testcase_params)                                                                                                                                                                     
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/siege/.cache/bazel/_bazel_siege/cb58467fc481c17711a9400b288e8730/sandbox/linux-sandbox/1/execroot/tensorflow_probability/bazel-out/k8-opt/bin/tensorflow_probability/python/experimen
tal/linalg/linear_operator_psd_kernel_test.runfiles/tensorflow_probability/tensorflow_probability/python/internal/test_combinations.py", line 303, in decorated
    execute_test_method()
  File "/usr/local/google/home/siege/.cache/bazel/_bazel_siege/cb58467fc481c17711a9400b288e8730/sandbox/linux-sandbox/1/execroot/tensorflow_probability/bazel-out/k8-opt/bin/tensorflow_probability/python/experimen
tal/linalg/linear_operator_psd_kernel_test.runfiles/tensorflow_probability/tensorflow_probability/python/internal/test_combinations.py", line 291, in execute_test_method
    test_method(**kwargs_to_pass)
  File "/usr/local/google/home/siege/.cache/bazel/_bazel_siege/cb58467fc481c17711a9400b288e8730/sandbox/linux-sandbox/1/execroot/tensorflow_probability/bazel-out/k8-opt/bin/tensorflow_probability/python/experimen
tal/linalg/linear_operator_psd_kernel_test.runfiles/tensorflow_probability/tensorflow_probability/python/experimental/linalg/linear_operator_psd_kernel_test.py", line 301, in test_matmul_grad_xla_kernelparams
    y, actuals, out_grad = f()
                           ^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 141, in error_handler
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 832, in __call__
    result = self._call(*args, **kwds)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 888, in _call
    self._initialize(args, kwds, add_initializers_to=initializers)
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 695, in _initialize
    self._concrete_variable_creation_fn = tracing_compilation.trace_function(
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 178, in trace_function
    concrete_function = _maybe_define_function(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 283, in _maybe_define_function
    concrete_function = _create_concrete_function(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 310, in _create_concrete_function
    traced_func_graph = func_graph_module.func_graph_from_py_func(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/framework/func_graph.py", line 1059, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 598, in wrapped_fn
    out = weak_wrapped_fn().__wrapped__(*args, **kwds)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py", line 52, in autograph_handler
    raise e.ag_error_metadata.to_exception(e)
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py", line 41, in autograph_handler
    return api.converted_call(
           ^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 439, in converted_call
    result = converted_f(*effective_args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/__autograph_generated_filej85tecr3.py", line 19, in tf__f
    actuals = ag__.converted_call(ag__.ld(tape).gradient, (ag__.ld(y), (ag__.ld(x1), ag__.ld(x2), ag__.ld(x), ag__.ld(kernel_args))), dict(output_gradients=ag__.ld(out_grad)), fscope)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 331, in converted_call
    return _call_unconverted(f, args, kwargs, options, False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 459, in _call_unconverted
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/backprop.py", line 1066, in gradient
    flat_grad = imperative_grad.imperative_grad(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/imperative_grad.py", line 67, in imperative_grad
    return pywrap_tfe.TFE_Py_TapeGradient(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/custom_gradient.py", line 506, in tape_grad_fn
    input_grads = grad_fn(*result_grads)
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/google/home/siege/.cache/bazel/_bazel_siege/cb58467fc481c17711a9400b288e8730/sandbox/linux-sandbox/1/execroot/tensorflow_probability/bazel-out/k8-opt/bin/tensorflow_probability/python/experimen
tal/linalg/linear_operator_psd_kernel_test.runfiles/tensorflow_probability/tensorflow_probability/python/experimental/linalg/linear_operator_psd_kernel.py", line 206, in grad_fn
    _, dx1, dx2, dx, dkernel_args = tf.while_loop(
                                    ^^^^^^^^^^^^^^                                                                                                                                                        [104/1112]
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/util/deprecation.py", line 660, in new_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/while_loop.py", line 241, in while_loop_v2
    return while_loop(
           ^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/while_loop.py", line 440, in while_loop
    return while_v2.while_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/while_v2.py", line 147, in while_loop
    cond_graph = func_graph_module.func_graph_from_py_func(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/framework/func_graph.py", line 997, in func_graph_from_py_func
    func_args, func_kwargs = _create_placeholders(args, kwargs, arg_names)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/framework/func_graph.py", line 1235, in _create_placeholders
    arg_trace_types = trace_type.from_value(tuple(args), signature_context)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 170, in from_value
    return default_types.Tuple(*(from_value(c, context) for c in value))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 170, in <genexpr>
    return default_types.Tuple(*(from_value(c, context) for c in value))
                                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 144, in from_value
    elif isinstance(value, trace.SupportsTracingProtocol):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/probability/python3.11/lib/python3.11/site-packages/typing_extensions.py", line 603, in __instancecheck__
    val = inspect.getattr_static(instance, attr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/inspect.py", line 1825, in getattr_static
    instance_result = _check_instance(obj, attr)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/inspect.py", line 1772, in _check_instance
    instance_dict = object.__getattribute__(obj, "__dict__")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: in user code:

    File "/usr/local/google/home/siege/.cache/bazel/_bazel_siege/cb58467fc481c17711a9400b288e8730/sandbox/linux-sandbox/1/execroot/tensorflow_probability/bazel-out/k8-opt/bin/tensorflow_probability/python/experim
ental/linalg/linear_operator_psd_kernel_test.runfiles/tensorflow_probability/tensorflow_probability/python/experimental/linalg/linear_operator_psd_kernel_test.py", line 297, in f  *
        actuals = tape.gradient(y, (x1, x2, x, kernel_args),
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/backprop.py", line 1066, in gradient  **
        flat_grad = imperative_grad.imperative_grad(
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/eager/imperative_grad.py", line 67, in imperative_grad
        return pywrap_tfe.TFE_Py_TapeGradient(
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/custom_gradient.py", line 506, in tape_grad_fn
        input_grads = grad_fn(*result_grads)
    File "/usr/local/google/home/siege/.cache/bazel/_bazel_siege/cb58467fc481c17711a9400b288e8730/sandbox/linux-sandbox/1/execroot/tensorflow_probability/bazel-out/k8-opt/bin/tensorflow_probability/python/experim
ental/linalg/linear_operator_psd_kernel_test.runfiles/tensorflow_probability/tensorflow_probability/python/experimental/linalg/linear_operator_psd_kernel.py", line 206, in grad_fn
        _, dx1, dx2, dx, dkernel_args = tf.while_loop(
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/util/deprecation.py", line 660, in new_func
        return func(*args, **kwargs)
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/while_loop.py", line 241, in while_loop_v2
        return while_loop(
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/while_loop.py", line 440, in while_loop
        return while_v2.while_loop(
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/ops/while_v2.py", line 147, in while_loop
        cond_graph = func_graph_module.func_graph_from_py_func(
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/framework/func_graph.py", line 997, in func_graph_from_py_func
        func_args, func_kwargs = _create_placeholders(args, kwargs, arg_names)
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/python/framework/func_graph.py", line 1235, in _create_placeholders
        arg_trace_types = trace_type.from_value(tuple(args), signature_context)
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 170, in from_value
        return default_types.Tuple(*(from_value(c, context) for c in value))
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 170, in <genexpr>
        return default_types.Tuple(*(from_value(c, context) for c in value))
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/tensorflow/core/function/trace_type/trace_type_builder.py", line 144, in from_value
        elif isinstance(value, trace.SupportsTracingProtocol):
    File "/tmp/probability/python3.11/lib/python3.11/site-packages/typing_extensions.py", line 603, in __instancecheck__
        val = inspect.getattr_static(instance, attr)
    File "/usr/lib/python3.11/inspect.py", line 1825, in getattr_static
        instance_result = _check_instance(obj, attr)
    File "/usr/lib/python3.11/inspect.py", line 1772, in _check_instance
        instance_dict = object.__getattribute__(obj, "__dict__")

    TypeError: this __dict__ descriptor does not support '_TupleWrapper' objects

NeilGirdhar · 2023-10-10T22:34:16Z

Looks like this is a problem with tensorflow? I think _TupleWrapper.__getattribute__ should be updated in some way to answer when the attribute name is __dict__.

Anyway, maybe the best thing to do is to skip the test for the time being, update the typing-extensions bound that blocks us users, and file a bug with TensorFlow?

ColCarroll · 2023-10-11T19:57:20Z

thanks for the reports!

this should be removed and pushed. tfp-nightly will get the update tomorrow.

Fixes #1753 See also tensorflow/tensorflow#60687 and tensorflow/tensorflow#61387 PiperOrigin-RevId: 572657594

piiq · 2023-11-28T16:40:05Z

Hi, I see there were a few new releases since the commit that solves this issue has been merged. Thanks a lot for the un-pin.

Do you know how long it usually takes for a new releases to make it to the colab environment?
Currently the colab environment comes with 0.22.0 pre-installed and it breaks colab as a platform for everyone who uses modern pydantic (which includes the majority of the llm tools).
Based on your experience how long might that take, and is there a way to speed it up?

ColCarroll · 2023-11-28T17:32:35Z

I think it is currently waiting on tensorflow-2.15 getting to colab -- I am not sure on that timing! tensorflow/tensorflow#62412 (comment) is possibly helpful to allow bumping TF and TFP.

Sorry for not being more helpful here.

piiq · 2023-11-28T17:40:46Z

Thanks a lot @ColCarroll
This is helpful enough
🙏☀️🌈

alexander-held mentioned this issue Oct 5, 2023

Relax typeguard version restriction after next Tensorflow release scikit-hep/cabinetry#428

Closed

ColCarroll mentioned this issue Oct 10, 2023

Pinning typing-extensions is broken on Python 3.12, Jax 0.4.18 #1759

Closed

NeilGirdhar mentioned this issue Oct 11, 2023

Internal definition breaks tensorflow-probability tensorflow/tensorflow#62092

Closed

copybara-service bot closed this as completed in 53c4dc6 Oct 11, 2023

jburnim pushed a commit that referenced this issue Oct 12, 2023

Remove pin on typing_extensions.

afe8cae

Fixes #1753 See also tensorflow/tensorflow#60687 and tensorflow/tensorflow#61387 PiperOrigin-RevId: 572657594

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is it possible to relax the dependency on `typing-extension < 4.6.0` #1753

Is it possible to relax the dependency on `typing-extension < 4.6.0` #1753

andresliszt commented Sep 27, 2023

csuter commented Sep 27, 2023

csuter commented Sep 27, 2023

alexander-held commented Oct 9, 2023

SiegeLordEx commented Oct 9, 2023

NeilGirdhar commented Oct 10, 2023 •

edited

Loading

ColCarroll commented Oct 11, 2023

piiq commented Nov 28, 2023 •

edited

Loading

ColCarroll commented Nov 28, 2023

piiq commented Nov 28, 2023

Is it possible to relax the dependency on typing-extension < 4.6.0 #1753

Is it possible to relax the dependency on typing-extension < 4.6.0 #1753

Comments

andresliszt commented Sep 27, 2023

csuter commented Sep 27, 2023

csuter commented Sep 27, 2023

alexander-held commented Oct 9, 2023

SiegeLordEx commented Oct 9, 2023

NeilGirdhar commented Oct 10, 2023 • edited Loading

ColCarroll commented Oct 11, 2023

piiq commented Nov 28, 2023 • edited Loading

ColCarroll commented Nov 28, 2023

piiq commented Nov 28, 2023

Is it possible to relax the dependency on `typing-extension < 4.6.0` #1753

Is it possible to relax the dependency on `typing-extension < 4.6.0` #1753

NeilGirdhar commented Oct 10, 2023 •

edited

Loading

piiq commented Nov 28, 2023 •

edited

Loading