Massive memory leak since 8.5.5 / leaking the entire stack frame and all local variables everywhere

[luk2302]

In #187 a change was implemented that included the current stack frame in an internal cache, therefore causing all local variables of the entire call stack to never be freed. Demo:

from dataclasses import dataclass
import marshmallow_dataclass
from marshmallow.schema import BaseSchema
from marshmallow_dataclass import _internal_class_schema


class Outer:
    def __del__(self):
        print("deleting outer...")


@dataclass(frozen=True)
class Sample:
    a: str
    def __del__(self):
        print("deleting sample...")


def inner_demo():
    schema = marshmallow_dataclass.class_schema(Sample, base_schema=BaseSchema)()
    data = schema.load({"a": "a string"})
    return data


def demo():
    d = inner_demo()
    o = Outer()


for i in range(10):
    demo()
    print(_internal_class_schema.cache_info())

print("done with demo, no object deleted so far")
_internal_class_schema.cache_clear()
print("cleared cache")

No __del__ is called on either Sample or Outer during the method invocations, only after the cache_clear can the local variables be garbage collected. If you run the demo with 8.4.4 or just replace _internal_class_schema(clazz, base_schema, clazz_frame) with _internal_class_schema(clazz, base_schema, None) in marshmallow_dataclass.class_schema. Additionally this also demonstrates that the caching does no longer work as there no cache hits at all anymore.

[LostInDarkMath]

Hi,
I came across this issue because our software has a memory leak in production. After days of troubleshooting and narrowing down the problem, I came across this ticket.

Are there any plans to fix this in the near future?
And how can it be that such a critical bug remains unfixed for 2 years?

[lovasoa]

I understand your frustration, but please refrain from "how can it be", when referring to free and open source software. Instead, you can offer time or money to free software projects to unblock situations like this.

[dairiki]

This is (mostly) fixed by #258 (marshmallow_dataclass >= 8.6.1).

When necessary, references to frames are still held until the schema is constructed (by accessing the .Schema attribute). Memory leakage is still possible if large numbers of marshmallow_dataclasses are constructed without accessing their .Schema attribute.

[vitaly-krugl]

Hi @dairiki - I wanted to follow up regarding:

Memory leakage is still possible if large numbers of marshmallow_dataclasses are constructed without accessing their .Schema attribute.

Does this mean that if a large number of these dataclasses are constructed using the dataclass constructor directly (without unmarshalling from data), then we still have a memory leak situation in the latest version of marshmallow-dataclass?

If so, what is the point of deferring schema construction, if deferring leads to serious issues? The deferring is also responsible for a terrible race condition in multi-threaded applications - see #282. Can the marshmallow_dataclass.dataclass decorator construct the schema inline instead (without deferring), and put all of these issues to rest?

Also, in this context, I am assuming that calling class_schema() on the dataclass is equivalent to accessing its .Schema attribute. Is this correct?

If I have a marshmallow dataclass and schema generation defined at top level of module like below, do I still need to be concerned about memory leaks after your fix?

@marshmallow_dataclass.dataclass
class MyDataclass:
  s: str
  i: int

MyDataclassSchema = marshmallow_dataclass.class_schema(MyDataclass)()

[vkruglik-aka]

Hi @lovasoa, sorry to bother you - could you please chime in on my comment above - #198 (comment)? Many thanks in advance.

[lovasoa]

Hi! I'm sorry, I don't use this package myself anymore and currently do not have a lot of bandwidth to maintain it. Happy to add new maintainers, though!

[vkruglik-aka]

Thank you for following up, @lovasoa. Has there been an attempt to get the core marshmallow project to take marshmallow-dataclass under its wing? They seem like a great pairing.