Fix build failure when shared_ptr<T> points to private std::enable_shared_from_this base

[rwgk]

Description

pybind11 v3 added std::enable_shared_from_this detection logic for shared_ptr<T>. When T privately inherits from std::enable_shared_from_this<T>, this causes a C++ compilation failure, even if T is unbound and never passed from Python.

This patch uses a SFINAE-based type_has_shared_from_this() overload to gracefully detect this case and preserve the behavior of the v2 release series.

The new test in test_shared_ptr.cpp,py

• Passes with pybind11 v2.13.6
• Fails with pybind11 master @ e7e5d6e (full error message)
• Passes with this PR.

---

The work on this PR was done with massive assistance from ChatGPT. Unfortunately I (rwgk) cannot share the link to the original conversation, but here is a PDF with most of it.

Initially I "just" wanted a reproducer, to explain a "known issue" in the pybind11 v3 Upgrade guide. This was tricky, because the code generating the original error was not accessible. It took about five iterations of trial-and-error before we (ChatGPT and I) had a reproducer (this took about 90 minutes). Then it took only two iterations (about 15 minutes) until we had the fix that is now in this PR.

---

A Changelog entry is not needed: This is a compatibility fix for an unreleased feature.

The Upgrade guide does no longer need to mention this!

---

Browse private_esft/manuscript tree

Browse private_esft/manuscript commits

[rwgk]

Compilation error for the test added with this PR, but without the fix in this PR:

g++ -o pybind11/tests/test_smart_ptr.os -c -std=c++20 -fPIC -fvisibility=hidden -O0 -g -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wundef -Wnon-virtual-dtor -Wunused-result -Werror -funsigned-char -Wpedantic -isystem /usr/include/python3.12 -isystem /usr/include/eigen3 -DPYBIND11_INTERNALS_VERSION=10000000 -DPYBIND11_TEST_BOOST -Ipybind11/include -Ipybind11/include pybind11/tests/test_smart_ptr.cpp
In file included from pybind11/include/pybind11/cast.h:16,
                 from pybind11/include/pybind11/attr.h:14,
                 from pybind11/include/pybind11/detail/class.h:12,
                 from pybind11/include/pybind11/pybind11.h:12,
                 from pybind11/include/pybind11/eval.h:14,
                 from pybind11/tests/pybind11_tests.h:3,
                 from pybind11/tests/constructor_stats.h:68,
                 from pybind11/tests/object.h:4,
                 from pybind11/tests/test_smart_ptr.cpp:11:
pybind11/include/pybind11/detail/type_caster_base.h: In instantiation of ‘std::shared_ptr<_Tp> pybind11::detail::smart_holder_type_caster_support::load_helper<T>::load_as_shared_ptr(void*, pybind11::handle) const [with T = test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT]’:
pybind11/include/pybind11/cast.h:981:67:   required from ‘pybind11::detail::copyable_holder_caster<type, std::shared_ptr<_Tp>, typename std::enable_if<pybind11::detail::copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled<type>::value, void>::type>::operator std::shared_ptr<_Tp>&() [with type = test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT; typename std::enable_if<pybind11::detail::copyable_holder_caster_shared_ptr_with_smart_holder_support_enabled<type>::value, void>::type = void]’
pybind11/include/pybind11/cast.h:54:47:   required from ‘typename pybind11::detail::make_caster<T>::cast_op_type<typename std::add_rvalue_reference<_Tp>::type> pybind11::detail::cast_op(make_caster<T>&&) [with T = const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&; typename make_caster<T>::cast_op_type<typename std::add_rvalue_reference<_Tp>::type> = std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&; make_caster<T> = type_caster<std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>, void>; typename std::add_rvalue_reference<_Tp>::type = const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&]’
pybind11/include/pybind11/cast.h:2069:51:   required from ‘Return pybind11::detail::argument_loader<Args>::call_impl(Func&&, std::index_sequence<Is ...>, Guard&&) && [with Return = void; Func = pybind11::detail::property_cpp_function_classic<test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT, std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> >::write<std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*>(std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*, const pybind11::handle&)::<lambda(test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&)>&; long unsigned int ...Is = {0, 1}; Guard = pybind11::detail::void_type; Args = {test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&}; std::index_sequence<Is ...> = std::integer_sequence<long unsigned int, 0, 1>]’
pybind11/include/pybind11/cast.h:2043:65:   required from ‘std::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> pybind11::detail::argument_loader<Args>::call(Func&&) && [with Return = void; Guard = pybind11::detail::void_type; Func = pybind11::detail::property_cpp_function_classic<test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT, std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> >::write<std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*>(std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*, const pybind11::handle&)::<lambda(test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&)>&; Args = {test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&}; std::enable_if_t<std::is_void<_Dummy>::value, pybind11::detail::void_type> = pybind11::detail::void_type]’
pybind11/include/pybind11/pybind11.h:425:78:   required from ‘void pybind11::cpp_function::initialize(Func&&, Return (*)(Args ...), const Extra& ...) [with Func = pybind11::detail::property_cpp_function_classic<test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT, std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> >::write<std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*>(std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*, const pybind11::handle&)::<lambda(test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&)>; Return = void; Args = {test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&}; Extra = {pybind11::is_method}]’
pybind11/include/pybind11/pybind11.h:277:19:   required from ‘pybind11::cpp_function::cpp_function(Func&&, const Extra& ...) [with Func = pybind11::detail::property_cpp_function_classic<test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT, std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> >::write<std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*>(std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*, const pybind11::handle&)::<lambda(test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT&, const std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>&)>; Extra = {pybind11::is_method}; <template-parameter-1-3> = void]’
pybind11/include/pybind11/pybind11.h:1768:16:   required from ‘static pybind11::cpp_function pybind11::detail::property_cpp_function_classic<T, D>::write(PM, const pybind11::handle&) [with PM = std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT> test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT::*; typename std::enable_if<std::is_member_pointer<PM>::value, int>::type <anonymous> = 0; T = test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT; D = std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>]’
pybind11/include/pybind11/pybind11.h:2165:59:   required from ‘pybind11::class_<type_, options>& pybind11::class_<type_, options>::def_readwrite(const char*, D C::*, const Extra& ...) [with C = test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT; D = std::shared_ptr<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>; Extra = {}; type_ = test_submodule_smart_ptr(pybind11::module_&)::ContainerUsingPrivateESFT; options = {}]’
pybind11/tests/test_smart_ptr.cpp:483:23:   required from here
pybind11/include/pybind11/detail/type_caster_base.h:794:64: error: ‘std::enable_shared_from_this<test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT>’ is an inaccessible base of ‘test_submodule_smart_ptr(pybind11::module_&)::PrivateESFT’
  794 |                 || !pybindit::memory::type_has_shared_from_this(type_raw_ptr)) {
      |                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~