Why using chunk cause memory leaking?

[lamkimphu258]

Version

• PHP 8.1.9
• Laravel 8.54

Hardware Specification

• CPU core i7-4600U
• RAM 16GB

Problem

I have a table in database which has nearly 7 millions record. For that reason, I need to write a command to sync data from MySQL to Elasticsearch to query faster. When I use chunk in my command, RAM usage is increased time by time and as you can guess, it is out of memory when I have not finished to sync data yet. Change to use Queue Job solve the problem.
I thought that chunk will load a chunk of data -> do something -> free memory -> load another chunk of data. But maybe, its step is load a chunk of data -> do something -> load another chunk of data. Why chunk doesn't free memory each time it finish a chunk of data and cause memory leaking?

[mikeydevelops]

Well as you can see in the source of the chunk method, it is unsetting the previous chunk before loading the next chunk. It is not actually freeing memory perse, it is only clearing what it loaded before your chunk code is run. That means somewhere in your code you are somehow storing the results in memory again and it is not automatically clearing on the next iteration. Maybe you are using a singleton class instance that stores the models inside it. If you would like to share the code you are using and we can try and debug it.

[lamkimphu258]

Hi @mikeydevelops, of course, this is my code, chunk size is 10_000, esService is elasticsearch service which has methods support elasticsearch in PHP:

Post:chunk(self::CHUNK_SIZE, function($posts) {
    $postIndexes = collect();
    foreach ($posts as $post) {
        $postIndexes->add($post->toArray());
    }
    $this->esService->bulkIndexing('posts', $postIndexes, 'id', true);
    unset($postIndexes);
}

[mikeydevelops]

You can simplify your code like this.

Post::chunk(self::CHUNK_SIZE, function($posts) {
    $this->esService->bulkIndexing('posts', $posts->map->toArray(), 'id', true);
});

But I couldn't find the source for esService, can you tell me which package are you using? maybe they are doing something that caches the records.
For lowering memory you can also try to lower CHUNK_SIZE too, because 10000 loaded models are quite a lot at once.

[lamkimphu258]

Thanks @mikeydevelops for your response. esService is class to handle operation of elasticsearch. This is implementattion of bulkIndexing:

public function bulkIndexing($index, $body, $keyId, $refresh)
{
    foreach ($body as $input) {
        if (is_null($keyId)) {
            $params['body'][] = [
                'index' => [
                    '_index' => $index,
                ],
            ];
        } else {
            $params['body'][] = [
                'index' => [
                    '_index' => $index,
                    '_id'    => $input[$keyId],
                ],
            ];
        }
        $params['body'][] = $input;
    }

    if ($refresh) {
        $params['refresh'] = true;
    }

    return $this->client->bulk($params);
}

Client in this case is Elastic\Elasticsearch\Client, my package I am using is php-elasticsearch