Hitting PHP memory limits while iterating over Drupal 8 EntityQuery results. How do I keep it down?

I have a D8 API endpoint that queries for a specific content type, applies any optional conditions, transforms the result to JSON, and returns to the client. I updated the PHP memory limit to 512M and I am still running into it. There are only 1500 records in Drupal so there really shouldnt be any reason why its so bad (341KB per record?!). If I just keep pumping up the memory to get it to run, the rendered JSON is less than 2 MB.

I know PHP garbage collection is automatic so I guess there are references that are being kept around.

I made several attempts to keep it down, like batching the queries, refactoring into functions, and explicitly calling gc_collect_cycles but nothing makes a difference.

How do I keep the memory consumption down while iterating over the results of a Drupal EntityQuery?

  protected function get() {
    echo "memory (start): " . memory_get_usage() . "\n<br>";

    //some setup and validation

    $query = $this->build_query($params);
    echo "memory (build_query): " . memory_get_usage() . "\n<br>";

    $results = $query->execute();
    echo "memory (execute): " . memory_get_usage() . "\n<br>";

    $items = [];

    $chunk_size = 50;
    $chunks = array_chunk(array_values($results), $chunk_size);
    echo "memory (chunk): " . memory_get_usage() . "\n<br>";

    foreach ($chunks as $chunk) {
      $items = array_merge($items, $this->load_nodes($chunk));
      echo "memory (chunk loaded): " . memory_get_usage() . "\n<br>";
    }
    echo "memory (all loaded): " . memory_get_usage() . "\n<br>";

    $response = [ 'results' => $items ];
    return new ResourceResponse($response);
  }

 

  protected function load_nodes($ids) {
    $items = [];
    $nodes = node_load_multiple($ids);
    foreach ($nodes as $node) {
      $items[] = $this->transform($node); 
    }
    return $items;
  }

 

  protected function transform($array) {
    $new = [
      "field1" => $array['field1'],
      "field2" => $array['field2'],
      //... for about 30 more fields, with some processing/manipulation ...
    ];
    return $new;
  }

And the output in regards to the memory echo is:

memory (start): 28297032
memory (build_query): 29984168
memory (execute): 31004048
memory (chunk): 31083864
memory (chunk loaded): 42175976
memory (chunk loaded): 50447792
memory (chunk loaded): 57609344
memory (chunk loaded): 66762688
memory (chunk loaded): 74555712
memory (chunk loaded): 86663016
memory (chunk loaded): 98514192
memory (chunk loaded): 110908336
memory (chunk loaded): 122792592
memory (chunk loaded): 134651328
memory (chunk loaded): 145622512
memory (chunk loaded): 156546072
memory (chunk loaded): 167805352
memory (chunk loaded): 178617040
memory (chunk loaded): 190400936
memory (chunk loaded): 201246256
memory (chunk loaded): 212387384
memory (chunk loaded): 223756088
memory (chunk loaded): 234898632
memory (chunk loaded): 246125624
memory (chunk loaded): 257136304
memory (chunk loaded): 268205304
memory (chunk loaded): 278744896
memory (chunk loaded): 289693184
memory (chunk loaded): 300491840
memory (chunk loaded): 310564624
memory (chunk loaded): 321204064
memory (chunk loaded): 333842760
memory (chunk loaded): 343723672
memory (chunk loaded): 344960728
memory (all loaded): 344960728

Shouldnt the memory consumption be going with each iteration of the chunk loading as the GC removes the old chunks? You'll notice that my endpoint only finishes up with 344 MB. The actually error is thrown somewhere in the Drupal core. Since I want to keep the max PHP memory at 128M, I still need to get my part of the memory down.

from Hitting PHP memory limits while iterating over Drupal 8 EntityQuery results. How do I keep it down?