[BUG]: std::weak_ptr expires for derived Python objects using py::smart_holder

[nsoblath]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

bc4a66d

Problem description

I have a C++ base class that I bind using the trampoline class setup and py::smart_holder. I extend the base class with a derived Python class. I also have a C++ class (and bound Python class) that holds a std::weak_ptr<base>. If I give the holder class a pointer to an instance of the derived Python class, passed with std::shared_ptr<base>, when I later check on the status of the weak pointer, I find that it's expired.

I've verified this on the latest commit to the master branch (bc4a66d) using both a Mac and running in a Linux container on the Mac. Please see PR #5624 and the code in this issue for buildable examples demonstrating the problem.

With the code below, the expected output is:

Creating a thing
Type: thing
Setting thing then printing
thing expired? 0
   type is thing
Printing after a return to python
thing expired? 0
   type is thing


Creating a derived thing
Type: DerivedThing
Setting thing then printing
thing expired? 0
   type is DerivedThing
Printing after a return to python
thing expired? 0
   type is DerivedThing

The actual output is:

Creating a thing
Type: thing
Setting thing then printing
thing expired? 0
   type is thing
Printing after a return to python
thing expired? 0
   type is thing


Creating a derived thing
Type: DerivedThing
Setting thing then printing
thing expired? 0
   type is DerivedThing
Printing after a return to python
thing expired? 1

The issue is the last line, which indicates that the weak_ptr is no longer aware that the object still exists.

I tested a these variations:

• Thing created in Python (first part of the included example) -- works
• DerivedThing created in Python (second part of the included example) -- does not work
• DerivedThing created in Python with no thing_trampoline -- works (except there's no virtual function overriding)
• DerivedThing created in Python where thing_trampoline also inherits from py::trampoline_self_life_support (commented out in the included example) -- does not work

Reproducible example code

C++ classes:

class thing : public std::enable_shared_from_this< thing >
{
    public:
        thing() = default;
        virtual ~thing() = default;

        virtual std::string get_type() const
        {
            return std::string("thing");
        }
};

class thing_holder
{
    public:
        thing_holder() = default;
        virtual ~thing_holder() = default;

        void print_thing() const
        {
            std::cout << "thing expired? " << f_thing.expired() << std::endl;
            if( ! f_thing.expired() )
            {
                std::cout << "   type is " << f_thing.lock()->get_type() << std::endl;
            }
            return;
        }

        void set_thing( std::shared_ptr< thing > a_thing_ptr )
        {
            f_thing = std::weak_ptr< thing >( a_thing_ptr );
            print_thing();  // print here to check the pointer while still in C++
            return;
        }

    private:
        std::weak_ptr< thing > f_thing;
};


Trampoline and Bindings:

class thing_trampoline : public thing //, public py::trampoline_self_life_support
{
    public:
        using thing::thing;

        std::string get_type() const override 
        {
            PYBIND11_OVERRIDE( std::string, thing, get_type, );
        }
};

std::list< std::string > export_thing( pybind11::module& mod )
{
    std::list< std::string > all_members;

    all_members.push_back( "Thing" );
    py::classh< thing, thing_trampoline >( mod, "Thing", "a thing" )
        .def( py::init< >() )
        .def( "get_type", &thing::get_type, "get the typename" )
        ;

    all_members.push_back( "ThingHolder" );
    py::class_< thing_holder >( mod, "ThingHolder", "holds things" )
        .def( py::init< >(), 
              py::return_value_policy::take_ownership )

        .def( "print_thing", &thing_holder::print_thing, "print thing" )
        .def( "set_thing", &thing_holder::set_thing, "set the thing" )
        ;

    return all_members;
}


Python class and executable script

class DerivedThing(dripline.core.Thing):
    def __init__(self):
        dripline.core.Thing.__init__(self)

    def get_type(self):
        return "DerivedThing"

if __name__ == '__main__':
    holder = thing_module.ThingHolder()

    print('Creating a thing')
    thing = thing_module.Thing()
    print(f'Type: {thing.get_type()}')
    
    print('Giving thing to the holder')
    holder.set_thing(thing)

    print('Printing after a return to python')
    holder.print_thing()

    print('\n')

    print('Creating a derived thing')
    derthing = DerivedThing()
    print(f'Type: {derthing.get_type()}')

    print('(Giving derived thing to the holder')
    holder.set_thing(derthing)

    print('Printing after a return to python')
    holder.print_thing()

Is this a regression? Put the last known working version here if it is.

Not a regression