Cannot get std::vector from binded member function having passed by reference std::vector as parameter

[MisesEnForce]

I have a dll Core, class __declspec(dllexport)'ing a certain class X. This class is in charge of some progressive calculation, successive values being kept track of in an "array". (I don't know yet which c++/python structure is the best for what I am trying to achieve.) This "array" can either be 1) a member of X or 2) returned by a member function of X. I am using this dll in another pyd dll, Binder, which as its name indicates, is in charge of doing python bindings of X (c++) member functions. Then I import the Binder.pyd dll in a python file, and I try to use python functions.

At the beginning, the "array" wasn't a member of X, which had a function void F(std::vector<double> & array) in charge of updating the array, but any run of the python file using the binded python version of F was triggering an access violation, precisely whenever resize or push_back where called on the std::vector<double>. I binded btw std::vector<double> as follows :

py::class_<std::vector<double>>(m, "stdvectordouble")
	.def(py::init<>())
	.def("clear", &std::vector<double>::clear)
	.def("pop_back", &std::vector<double>::pop_back)
	.def("push_back", (void (std::vector<double>::*)(const double &)) &std::vector<double>::push_back)
	.def("__len__", [](const std::vector<double> &v) { return v.size(); })
	.def("__iter__", [](std::vector<double> &v) {
            return py::make_iterator(v.begin(), v.end());
}, py::keep_alive<0, 1>());

I tried std::share_ptr<double> + int (size) instead, same type of issue as well. I suspect that my problem is ultimately caused by this :

https://support.microsoft.com/en-us/help/172396/you-may-experience-an-access-violation-when-you-access-an-stl-object-t

though I don't really know how to put the proposed solution in practice in my case.

So that I have at the switch to keeping the "array" as a member of X, but I really don't know what the better structure to give to it to by able to easily bind its get function so that I coudl extract the "array" in my python code. (I would also like to have a c++ code recyclable enough in other c++ contexts, that is : I don't want to have the signature of F to be too much pybind11-dependent.)

Remark : at the origin, Core and Binder dll's were one single dll, and I wasn't having the acces violation error triggered at run time.

[jagerman]

In this specific case it would probably be a good idea to use pybind11/stl_bind.h's bind_vector (see the docs). It provides a python-like wrapper (e.g. append rather than push_back) around a C++ vector type, with other nice goodies like slicing support.

(It looks like an omission that we don't have a clear method, which Python 3.3+ lists do have. You can add additional definitions to the class_<...> object returned by bind_vector, of course, but adding clear would be an easy PR, if you're up for it).

As to the actual issue, one possible solution is avoid binding stl functions directly, but instead use thin lambda functions that wrap the stl functions rather than directly binding the stl functions. e.g. .def("clear", [](std::vector<double> &v) { v.clear(); }).

[MisesEnForce]

Hum, replacing the above py::class_<std::vector<double>>(m, "stdvectordouble") code by py::bind_vector<std::vector<double>>(m, "stdvectordouble"); as advised in the link you pointed to led to the same error : "Exception thrown at 0x0F5C67A8 (Core.dll) in python.exe: 0xC0000005: Access violation writing location 0x00000000." triggered at the first line where the vector was modified in the c++.

[jagerman]

Did you remove all the other .defs as well? That said, that location 0x00000000 looks a bit odd: as if something isn't initialized (and is instead set to nullptr).

[MisesEnForce]

I replaced

py::class_<std::vector<double>>(m, "stdvectordouble")
	.def(py::init<>())
	.def("clear", &std::vector<double>::clear)
	.def("pop_back", &std::vector<double>::pop_back)
	.def("push_back", (void (std::vector<double>::*)(const double &)) &std::vector<double>::push_back)
	.def("__len__", [](const std::vector<double> &v) { return v.size(); })
	.def("__iter__", [](std::vector<double> &v)
					   {
							return py::make_iterator(v.begin(), v.end());
					   }
		,
		py::keep_alive<0, 1>() //Keep vector alive while iterator is used 
	)
	.def("clear", [](std::vector<double> &v) { v.clear(); })
	;

with py::bind_vector<std::vector<double>>(m, "stdvectordouble"); I have the python calling the c++ and a c++ unit test calling it also. I debugged the vector (default initizalized) in both cases : in the c++ unit test case (where I don't have access violation) the capacity of the vector is zero, as expected, and is rightfully changed after a reserve on it ; in the python case the capacity is equal to 532660159, even after a reserve on it ... That's the only difference at debug, and it's strange.

[MisesEnForce]

I added a .def("reserve0", [](std::vector<double> &v) { v.reserve(0); }) and I call reserve0 after having declared the stdvectordouble and I still see the std::vector<double> at the entry of the c++ having a capacity equal to 532660159.
The rationale was to enforce the vector to have a 0 capacity at the entry of the c++, but it is still not working.

[MisesEnForce]

Setting in the c++ my vector equal to std::vector<double>() allows me to modify it, finally. This is EXTREMELY strange. I still have

after execution of the code, which makes me thing I have so ownership issue with my vector ...

[jagerman]

Some other possibilities:

You mentioned in the OP:

Remark : at the origin, Core and Binder dll's were one single dll, and I wasn't having the acces violation error triggered at run time.

which makes me think there may be an issue here with compiler settings, i.e. the two dlls are compiled with incompatible settings. Are there different compiler flags, maybe different C++ standards used across the two dlls? (I'm not the best to help if that's the case—my Windows compiler experience is pretty limited—but might be a direction to look at).

Some other, perhaps less likely ideas:

Are you inheriting from the class, or just using it directly? In case you are inheriting, are you remembering to call stdvectordouble.__init__(self) from the Python subclass constructor?

There could be an issue with the compiler used for Python vs the compiler you are using. (This is a sort of variation of my first point, but might also be an issue).

[MisesEnForce]

I use the class directly, and I posted its code above. (Which was working before, as mentionned. Meaning by that : the vector had a 0 capacity at the enter of the c++.)

Compilers seem to be ok also.

Hum ... Maybe this exception (and associated call stack) could help :

[MisesEnForce]

Ok, as I told, I do a convergence = Binder.stdvectordouble() where Binder is the python module I produce with pybind11, and right after I do a convergence.printv() where I binded c++ code printing size, capacity and (if any) elements of the underlying vector. I see it having a zero capacity. Then I call MyBinderObject.PRICE(NbPaths, convergence) in python, and immediately at the beginning of the c++ PRICE function, I see the underlying vector having a capacity equal to 532660159 as mentionned above.

In hexadecimal 532660159 is written 0xdeadbeef and it appears in d:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.12.25827\include\xhash in #define _HASH_SEED (size_t)0xdeadbeef. Looking for _HASH_SEED shows that it's used :

in the function template<class _Kty> inline size_t hash_value(const _Kty& _Keyval). And this function appears in a comment in c:\PYTHON\PYTHON_OFFICIAL\python-3.6.3\include\abstract.h :

Looking for long PyObject_Hash(PyObject *o); finally reveals a PyAPI_FUNC(Py_hash_t) PyObject_HashNotImplemented(PyObject *); in c:\PYTHON\PYTHON_OFFICIAL\python-3.6.3\include\object.h and this latter function is used in c:\PYTHON\pybind11\include\pybind11\pytypes.h as follows :

[fwjavox]

It looks like an omission that we don't have a clear method, which Python 3.3+ lists do have. You can add additional definitions to the class_<...> object returned by bind_vector, of course, but adding clear would be an easy PR, if you're up for it

I also stumbled about the missing clear method, so I made a pull request to add it: #2074