Question/ issue: potential memory leak in to_bytes_packed()?

[imprs]

Hi, thank you for your great work on pycapnp!

I am encountering an issue on an embedded device that publishes images at a fixed FPS. In my code, after creating a message, I noticed that using to_bytes_packed() causes the memory consumption to steadily increase until the device eventually runs out of memory, requiring a restart. Interestingly, this does not happen when using to_bytes().

I am wondering if there’s anything specific about the packed serialization that could explain this behavior.

Here is a quick example to reproduce the issue:

Python: 3.11.11
pycapnp: 2.0.0

schema byte_image.capnp

struct ByteImage {
  timestamp @0 :UInt64;

  height @1 :UInt32;
  width @2 :UInt32;

  mode @3 :Text;
  compression @4 :Text = "none";

  data @5 :Data;
}

test.py

import argparse
import numpy as np
import cv2
from datatypes_python import ByteImage
from memory_profiler import profile

def create_message():
    image = np.random.randint(low=0, high=255, size=(640, 420, 3), dtype=np.uint8)
    message = ByteImage.new_message()
    message.data = (cv2.imencode('.jpg', image)[1]).tobytes()

    return message

@profile
def test_to_bytes(runs):
    for i in range(runs):
        message = create_message()
        msg_bytes = message.to_bytes()

@profile
def test_to_bytes_packed(runs):
    for i in range(runs):
        message = create_message()      
        msg_bytes = message.to_bytes_packed()

if __name__ == "__main__":
    parser = argparse.ArgumentParser("Test pycapnp bytes packed")
    parser.add_argument("--runs", type=int, default=100, help="Number of runs")
    args = parser.parse_args()

    test_to_bytes(args.runs)
    test_to_bytes_packed(args.runs)

Logs:

1. 100 runs

python test.py --runs 100

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    14     63.4 MiB     63.4 MiB           1   @profile
    15                                         def test_to_bytes(runs):
    16     71.1 MiB    -92.8 MiB         101       for i in range(runs):
    17     71.1 MiB    -65.4 MiB         100           message = create_message()
    18     71.1 MiB    -92.9 MiB         100           msg_bytes = message.to_bytes()


Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    20     68.4 MiB     68.4 MiB           1   @profile
    21                                         def test_to_bytes_packed(runs):
    22    101.4 MiB     -0.4 MiB         101       for i in range(runs):
    23    101.2 MiB     26.8 MiB         100           message = create_message()      
    24    101.4 MiB      5.0 MiB         100           msg_bytes = message.to_bytes_packed()

2. 150 runs:

python test.py --runs 150

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    14     63.5 MiB     63.5 MiB           1   @profile
    15                                         def test_to_bytes(runs):
    16     71.2 MiB   -133.2 MiB         151       for i in range(runs):
    17     71.2 MiB   -100.5 MiB         150           message = create_message()
    18     71.2 MiB   -133.0 MiB         150           msg_bytes = message.to_bytes()


Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    20     69.4 MiB     69.4 MiB           1   @profile
    21                                         def test_to_bytes_packed(runs):
    22    118.0 MiB     -0.3 MiB         151       for i in range(runs):
    23    118.0 MiB     23.7 MiB         150           message = create_message()      
    24    118.0 MiB     23.7 MiB         150           msg_bytes = message.to_bytes_packed()

3. 200 runs:

python test.py --runs 200

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    14     63.3 MiB     63.3 MiB           1   @profile
    15                                         def test_to_bytes(runs):
    16     71.2 MiB   -196.3 MiB         201       for i in range(runs):
    17     71.2 MiB   -144.8 MiB         200           message = create_message()
    18     71.2 MiB   -196.1 MiB         200           msg_bytes = message.to_bytes()


Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    20     69.3 MiB     69.3 MiB           1   @profile
    21                                         def test_to_bytes_packed(runs):
    22    133.5 MiB     -0.9 MiB         201       for i in range(runs):
    23    133.5 MiB     29.5 MiB         200           message = create_message()      
    24    133.5 MiB     31.5 MiB         200           msg_bytes = message.to_bytes_packed()

For to_bytes(), Mem usage stays roughly the same; however, for to_bytes_packed(), it keeps increasing.