fail to collect references in release build

[mgood7123]

jump to #259 (comment)

i am unsure how to go about keeping roots alive as they get eligible for garbage collection immediately after creation

    ManagedObjState state;

    managed_obj_init(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE);
    
    managed_obj_t p = managed_obj_make(&state, malloc(500));
    printf("p is %p\n", p);

    // managed_obj_describe(&state);

    printf("first collect, p should not be collected\n");
    managed_obj_collect(&state); // should not collect p

    printf("if p has been collected then we have not managed to track it correctly\n");
    p = (managed_obj_t) 0x6; // no longer referenced

    printf("second collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("if p has NOT been collected then we have not managed to track it correctly\n");

    managed_obj_deinit(&state);

(i assume there would be more that needs to be done in my implementation (below) to correctly support the above tho i am unsure as to what would be needed)

first collect, p should not be collected

object 0x7ffbf7100000 with pointer 0x12ea080 is being freed.
Total Collection duration: 0 seconds, 0 milliseconds, 192 microseconds

if p has been collected then we have not managed to track it correctly

second collect, p should be collected

Total Collection duration: 0 seconds, 0 milliseconds, 70 microseconds

if p has NOT been collected then we have not managed to track it correctly

deinit
deinit - collect
Total Collection duration: 0 seconds, 0 milliseconds, 43 microseconds
deinit - shutdown

the complete .h and .c file is https://gist.github.com/mgood7123/8a25971c90faa61397701361a0dd8be5

[mgood7123]

welp i figured out a way, tho it is not scalable

    mps_root_t static_stack_root;
    mps_addr_t static_stack[200];
    int static_stack_counter;

    mps_res_t res = mps_reserve(&state->static_stack[state->static_stack_counter], state->ap, size);
    if (res != MPS_RES_OK) managed_obj_error("out of memory in make_pointer");
    // printf("reserved pointer %p\n", static_stack[static_stack_counter]);
    obj = (managed_obj_t)state->static_stack[state->static_stack_counter];
    obj->pointer.type = MANAGED_OBJECT_TYPE_POINTER;
    obj->pointer.pointer = pointer;
    // printf("pointer %p contains pointer %p\n", obj, obj->pointer.pointer);
  } while(!mps_commit(state->ap, state->static_stack[state->static_stack_counter], size));
  state->static_stack_counter++;
  if (state->static_stack_counter == 200)
    state->static_stack_counter = 0;

    /* Create an MPS arena.  There is usually only one of these in a process.
        It holds all the MPS "global" state and is where everything happens. */
    MPS_ARGS_BEGIN(args) {
        MPS_ARGS_ADD(args, MPS_KEY_ARENA_SIZE, arena_size);
        res = mps_arena_create_k(&state->arena, mps_arena_class_vm(), args);
    } MPS_ARGS_END(args);
    if (res != MPS_RES_OK) managed_obj_error("Couldn't create arena");

// ...

  res = mps_root_create_area(&state->static_stack_root, state->arena, mps_rank_exact(), 0,
                             state->static_stack, state->static_stack + 200, mps_scan_area, NULL);
  if(res != MPS_RES_OK) managed_obj_error("Couldn't register stack root");

void managed_obj_deinit(ManagedObjState * state) {
    printf("deinit\n");

    memset(state->static_stack, 0, sizeof(mps_addr_t)*200);

    printf("deinit - collect\n");
    managed_obj_collect(state);

    printf("deinit - shutdown\n");
    mps_arena_park(state->arena);        /* ensure no collection is running */
    mps_root_destroy(state->static_stack_root); /* destroy static stack root */
    // mps_root_destroy(state->stack_root); /* destroy stack root before thread */
    // mps_thread_dereg(state->thread);     /* deregister thread before ap */
    mps_ap_destroy(state->ap);       /* destroy ap before pool */
    mps_pool_destroy(state->pool);   /* destroy pool before fmt */
    mps_fmt_destroy(state->fmt);     /* destroy fmt before chain */
    mps_chain_destroy(state->chain); /* destroy chain before arena */
    mps_arena_destroy(state->arena);     /* last of all */

    memset(state, 0, sizeof(ManagedObjState));
}

[mgood7123]

is there a way we can make this more scalable to larger amounts of allocations?

[mgood7123]

how feasible would it be to just register and destroy gc root structures as they are passed around?

like 1 root for each gc allocation being tracked

[rptb1]

Hello!

We've look at your code, and it has a problem with the registration of the stack root. Your are registering the stack from within a function, and using that function's stack frame as the cold end of the stack. See https://gist.github.com/mgood7123/8a25971c90faa61397701361a0dd8be5#file-managed_object_obj-c-L434

This means that none of the functions calling managed_obj_make will have their stack frames included in the root, and objects they point to might be GC'd by the MPS.

The documentation section Thread Roots describes how the stack root must be created. All functions that have roots in registers must be called by the function that locates the cold end in the way shown.

In general, you do not need to create and destroy roots very often -- certainly not once per object. The MPS is designed so that you register roots, then store pointers to objects you want to keep alive in those roots. The typical case is that the roots are the stack and registers of your thread, and all objects (recursively) reachable from there are kept alive. Preserving objects is just a matter of storing pointers to memory.

You definitely could have a kind of C++ "root object" that had a call to mps_root_create in its constructor and mps_root_destroy in its destructor, but that's the sort of thing we expect you do rarely: typically only once per process run.

We can't quite see from your code what you're trying to achieve. Feel free to explain your application. We're curious and would like to help.

Also we recommend you study the Scheme Example to see how it creates roots.

There might be a misunderstanding of what we call a "root". See https://memory-pool-system.readthedocs.io/en/version-1.118/glossary/r.html#term-root . A root is not GC'd by the MPS. A root is a (set of) pointers to objects managed by the MPS. So your original question "how to keep a root alive" might be trying to ask "how to keep an object referenced by a root alive"? The answer is that the MPS preserves all objects referenced by roots. If an object is dying, it is not referenced by a root. That could be because of the cold end problem I mentioned above.

Thanks.

[mgood7123]

i tried using the stack bottom (GC_get_stack_base() from boehm) as the thread stack root

which works correctly in debug build

but fails in a release build (no GC takes place)

    res = mps_thread_reg(&state->thread, state->arena);
    if (res != MPS_RES_OK) managed_obj_error("Couldn't register thread");

    struct MANAGED_STACK_ADDRESS_BOEHM_GC_stack_base sb_end;
    memset(&sb_end, 0, sizeof(struct MANAGED_STACK_ADDRESS_BOEHM_GC_stack_base));

    int gcres = MANAGED_STACK_ADDRESS_BOEHM_GC_get_stack_base(&sb_end);
    if (gcres == MANAGED_STACK_ADDRESS_BOEHM_GC_SUCCESS) {
    } else if (gcres == MANAGED_STACK_ADDRESS_BOEHM_GC_UNIMPLEMENTED) {
      managed_obj_error("GC_get_stack_base(start) unimplemented\n");
    } else {
      managed_obj_error("GC_get_stack_base(start) failed: %d\n", gcres);
    }

    printf("thread stack bottom: %p\n", sb_end.mem_base);

    // res = mps_root_create_thread(&state->thread_stack_root, state->arena, state->thread, (void*)sb_end.mem_base);
    // if (res != MPS_RES_OK) managed_obj_error("Couldn't create stack root");

    res = mps_root_create_reg(&state->thread_stack_root, state->arena, mps_rank_ambig(), 0, state->thread, mps_stack_scan_ambig, sb_end.mem_base, 0);
    if (res != MPS_RES_OK) printf("Couldn't create root");

[mgood7123]

Hello!

We've look at your code, and it has a problem with the registration of the stack root. Your are registering the stack from within a function, and using that function's stack frame as the cold end of the stack. See https://gist.github.com/mgood7123/8a25971c90faa61397701361a0dd8be5#file-managed_object_obj-c-L434

This means that none of the functions calling managed_obj_make will have their stack frames included in the root, and objects they point to might be GC'd by the MPS.

The documentation section Thread Roots describes how the stack root must be created. All functions that have roots in registers must be called by the function that locates the cold end in the way shown.

In general, you do not need to create and destroy roots very often -- certainly not once per object. The MPS is designed so that you register roots, then store pointers to objects you want to keep alive in those roots. The typical case is that the roots are the stack and registers of your thread, and all objects (recursively) reachable from there are kept alive. Preserving objects is just a matter of storing pointers to memory.

You definitely could have a kind of C++ "root object" that had a call to mps_root_create in its constructor and mps_root_destroy in its destructor, but that's the sort of thing we expect you do rarely: typically only once per process run.

We can't quite see from your code what you're trying to achieve. Feel free to explain your application. We're curious and would like to help.

Also we recommend you study the Scheme Example to see how it creates roots.

There might be a misunderstanding of what we call a "root". See https://memory-pool-system.readthedocs.io/en/version-1.118/glossary/r.html#term-root . A root is not GC'd by the MPS. A root is a (set of) pointers to objects managed by the MPS. So your original question "how to keep a root alive" might be trying to ask "how to keep an object referenced by a root alive"? The answer is that the MPS preserves all objects referenced by roots. If an object is dying, it is not referenced by a root. That could be because of the cold end problem I mentioned above.

Thanks.

ultimately i would like to create an api with MPS that allows me to basically have a stateful GC object that can "attach" and "detach" from threads

for example

// data is shared between T1 T2 and T3
//
mps_gc_1 for threads 1 2 3

// data is shared between T2 T4 and T5
//
// NOTE: mps_gc_1 will not touch data allocated by mps_gc_2 even though both are attached to thread 2
//
mps_gc_2 for threads 4 2 5

// dedicated MPS GC for thread 7, data is known not to be shared between any other thread
//
//   it is possible to localize data by carefully attaching and detaching from threads as data moves between threads
//
mps_gc_3 for thread 7

mps_gc 1 and 2 can co-exist with mps_gc 3 due to them running on seperate threads
unlike a global gc which runs on all threads even if data only exists on a single thread

this should co-exist with other GC's as long as we use a stop-the-world approach

[mgood7123]

i tried

    ManagedObjState state;

    managed_obj_init(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE);

void managed_obj_init(ManagedObjState * state, size_t arena_size) {
    if (state == NULL) {
        managed_obj_error("state cannot be null");
    }
    memset(state, 0, sizeof(ManagedObjState));

    mps_res_t res;

// ...

/*

        A risk of using tagged pointers in registers and on the stack
        is that in some circumstances, an optimizing compiler might
        optimize away the tagged pointer, keeping only the untagged
        version of the pointer. In this situation the pointer would be
        ignored and if it was the last reference to the object the MPS
        might incorrectly determine that it was dead. 

        #. Choose to tag pointers with zero, setting ``scan_area`` to
           :c:func:`mps_scan_area_tagged` and setting ``pattern`` to
           zero.

*/

    void *marker = state;

    res = mps_root_create_thread_tagged(
      &state->thread_stack_root, state->arena,
      mps_rank_ambig(), (mps_rm_t)0,
      state->thread, mps_scan_area_tagged,
      sizeof(mps_word_t) - 1, (mps_word_t)0,
      marker
    );

    if (res != MPS_RES_OK) printf("Couldn't create root");
}

but release build still fails to find the pointers where a debug build would successfully find the pointers

void managed_obj_init(ManagedObjState * state, size_t arena_size) {
    if (state == NULL) {
        managed_obj_error("state cannot be null");
    }
    memset(state, 0, sizeof(ManagedObjState));

    mps_res_t res;

    MPS_ARGS_BEGIN(args) {
        MPS_ARGS_ADD(args, MPS_KEY_ARENA_SIZE, arena_size);
        res = mps_arena_create_k(&state->arena, mps_arena_class_vm(), args);
    } MPS_ARGS_END(args);
    if (res != MPS_RES_OK) managed_obj_error("Couldn't create arena");

  
    /* Make sure we can pick up finalization messages. */
    mps_message_type_enable(state->arena, mps_message_type_finalization());

  MPS_ARGS_BEGIN(args) {
    MPS_ARGS_ADD(args, MPS_KEY_FMT_ALIGN, MANAGED_OBJECT_ALIGNMENT);
    MPS_ARGS_ADD(args, MPS_KEY_FMT_SCAN, managed_obj_scan);
    MPS_ARGS_ADD(args, MPS_KEY_FMT_SKIP, managed_obj_skip);
    MPS_ARGS_ADD(args, MPS_KEY_FMT_FWD, managed_obj_fwd);
    MPS_ARGS_ADD(args, MPS_KEY_FMT_ISFWD, managed_obj_isfwd);
    MPS_ARGS_ADD(args, MPS_KEY_FMT_PAD, managed_obj_pad);
    res = mps_fmt_create_k(&state->fmt, state->arena, args);
  } MPS_ARGS_END(args);
  if (res != MPS_RES_OK) managed_obj_error("Couldn't create obj format");

  res = mps_chain_create(&state->chain,
                         state->arena,
                         (sizeof(managed_obj_gen_params) / sizeof(managed_obj_gen_params[0])),
                         managed_obj_gen_params);
  if (res != MPS_RES_OK) managed_obj_error("Couldn't create obj chain");

  MPS_ARGS_BEGIN(args) {
    MPS_ARGS_ADD(args, MPS_KEY_CHAIN, state->chain);
    MPS_ARGS_ADD(args, MPS_KEY_FORMAT, state->fmt);
    res = mps_pool_create_k(&state->pool, state->arena, mps_class_amc(), args);
  } MPS_ARGS_END(args);
  if (res != MPS_RES_OK) managed_obj_error("Couldn't create obj pool");

  res = mps_ap_create_k(&state->ap, state->pool, mps_args_none);
  if (res != MPS_RES_OK) managed_obj_error("Couldn't create obj allocation point");

    void *marker = state;

    res = mps_root_create_thread_tagged(
      &state->thread_stack_root, state->arena,
      mps_rank_ambig(), (mps_rm_t)0,
      state->thread, mps_scan_area_tagged,
      sizeof(mps_word_t) - 1, (mps_word_t)0,
      marker
    );

    if (res != MPS_RES_OK) printf("Couldn't create root");
}

managed_obj_t managed_obj_make(ManagedObjState * state, void * pointer)
{
  mps_addr_t port_ref;
  managed_obj_t obj;
  size_t size = MANAGED_OBJECT_ALIGN_OBJ(sizeof(managed_obj_pointer_s));
  do {
    mps_res_t res = mps_reserve(&state->static_stack[state->static_stack_counter], state->ap, size);
    if (res != MPS_RES_OK) managed_obj_error("out of memory in make_pointer");
    obj = (managed_obj_t)state->static_stack[state->static_stack_counter];
    obj->pointer.type = MANAGED_OBJECT_TYPE_POINTER;
    obj->pointer.pointer = pointer;
  } while(!mps_commit(state->ap, state->static_stack[state->static_stack_counter], size));
  printf("reserved and comitted object %p with pointer %p\n", obj, obj->pointer.pointer);
  state->static_stack_counter++;
  if (state->static_stack_counter == 200)
    state->static_stack_counter = 0;
  state->total += sizeof(managed_obj_pointer_s);
  port_ref = obj;
  mps_finalize(state->arena, &port_ref);
  return obj;
}

void managed_obj_collect(ManagedObjState * state) {
    unsigned long collect_start = mps_clock();

    mps_arena_collect(state->arena);
    managed_obj_mps_chat(state);

    unsigned long collect_end = mps_clock();

    ManagedObjTime collect_duration_time = managed_obj_convert_to_time(collect_end - collect_start);

    printf("Total Collection duration: %d seconds, %d milliseconds, %d microseconds\n", collect_duration_time.seconds, collect_duration_time.milliseconds, collect_duration_time.microseconds);
}

void managed_obj_deinit(ManagedObjState * state) {
    printf("deinit\n");

    memset(state->static_stack, 0, sizeof(mps_addr_t)*200);

    printf("deinit - collect after memset\n");
    managed_obj_collect(state);
    printf("deinit - collected after memset\n");

    printf("deinit - shutdown\n");
    mps_arena_park(state->arena);        /* ensure no collection is running */
    mps_root_destroy(state->thread_stack_root); /* destroy stack root before thread */
    mps_thread_dereg(state->thread);     /* deregister thread before ap */
    mps_ap_destroy(state->ap);       /* destroy ap before pool */
    mps_pool_destroy(state->pool);   /* destroy pool before fmt */
    mps_fmt_destroy(state->fmt);     /* destroy fmt before chain */
    mps_chain_destroy(state->chain); /* destroy chain before arena */
    mps_arena_destroy(state->arena);     /* last of all */

    memset(state, 0, sizeof(ManagedObjState));
}

TEST(MPS_collect2, 1) {
    ManagedObjState state;

    managed_obj_init(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE);
    
    managed_obj_t p1 = managed_obj_make(&state, malloc(500));
    managed_obj_t p2 = managed_obj_make(&state, malloc(500));
    printf("p1 is %p, &p = %p\n", p1, &p1);
    printf("p2 is %p, &p = %p\n", p2, &p2);

    printf("first collect, p should not be collected\n");
    managed_obj_collect(&state); // should not collect p

    printf("if p has been collected then we have not managed to track it correctly\n");
    p1 = (managed_obj_t) 0x6; // no longer referenced
    p2 = (managed_obj_t) 0x6; // no longer referenced

    printf("second collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("if p has NOT been collected then we have not managed to track it correctly\n");

    managed_obj_deinit(&state);
}

[mgood7123]

if we build using gcc and g++ instead of clang and clang++ then a release build successfully finds our pointers and GC's them

in clang/clang++ this seems to happen at -O2 and higher

using https://github.com/Ravenbrook/mps/master

clang and clang++

testBuilder_add_include(gc_mps mps_master/code)
testBuilder_add_include(gc_mps include)
testBuilder_add_source(gc_mps mps_master/code/mps.c)
testBuilder_add_source(gc_mps src/managed_object_obj.c)
# CONFIG_VAR_COOL
# CONFIG_VAR_HOT
# CONFIG_VAR_RASH
testBuilder_add_compile_option(gc_mps "SHELL:-DCONFIG_VAR_COOL")
testBuilder_build_static_library(gc_mps)

valgrind output (debug) https://gist.github.com/mgood7123/a565343e2f0f26c2f49ca6849632ab5c

valgrind output (release) https://gist.github.com/mgood7123/008d9dd435cf8660dd305449ceec5331

debug build, variant COLD

[ RUN      ] MPS_collect2.1
reserved and comitted object 0x7fc3f691a000 with pointer 0xd7c1c0
reserved and comitted object 0x7fc3f691a018 with pointer 0xd7c3c0
p1 is 0x7fc3f691a000, &p = 0x7fff73cc7228
p2 is 0x7fc3f691a018, &p = 0x7fff73cc7220
first collect, p should not be collected
Total Collection duration: 0 seconds, 0 milliseconds, 554 microseconds
if p has been collected then we have not managed to track it correctly
second collect, p should be collected
object 0x7fc3f6980000 with pointer 0xd7c1c0 is being freed.
object 0x7fc3f6980018 with pointer 0xd7c3c0 is being freed.
Total Collection duration: 0 seconds, 1 milliseconds, 557 microseconds
if p has NOT been collected then we have not managed to track it correctly
deinit
deinit - collect after memset
Total Collection duration: 0 seconds, 0 milliseconds, 634 microseconds
deinit - collected after memset
deinit - shutdown
[       OK ] MPS_collect2.1 (16 ms)

release build, variant COLD

[ RUN      ] MPS_collect2.1
reserved and comitted object 0x7f4da271a000 with pointer 0x1b911c0
reserved and comitted object 0x7f4da271a018 with pointer 0x1b913c0
p1 is 0x7f4da271a000, &p = 0x7fff2f5e4700
p2 is 0x7f4da271a018, &p = 0x7fff2f5e46f8
first collect, p should not be collected
Total Collection duration: 0 seconds, 0 milliseconds, 83 microseconds
if p has been collected then we have not managed to track it correctly
second collect, p should be collected
Total Collection duration: 0 seconds, 0 milliseconds, 29 microseconds
if p has NOT been collected then we have not managed to track it correctly
deinit
deinit - collect after memset
Total Collection duration: 0 seconds, 0 milliseconds, 58 microseconds
deinit - collected after memset
deinit - shutdown
[       OK ] MPS_collect2.1 (1 ms)

[mgood7123]

if i make my pointer volatile then it gets collected at -O2 and -O3 but i cannot force other libraries to use volatile

    volatile managed_obj_t p1 = managed_obj_make(&state, malloc(500));
    volatile managed_obj_t p2 = managed_obj_make(&state, malloc(500));

// ...

    p1 = (managed_obj_t) 0x6; // no longer referenced
    p2 = (managed_obj_t) 0x6; // no longer referenced

// ...

    printf("second collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

[mgood7123]

this seems like with optimizations, attempting to bring a variable into the local stack of a function MAY bind it to the frame, resulting in it being uncollectable until the frame ends

this in turn CAN be troublesome if we attempt to init and destroy the GC in the same frame, since the GC cannot reclaim a variable that is bound to the same frame as it (or even may leak to a parent frame variable)
(leaking would technically be UB as it would yield a reference that exists but is untracked since the GC has been destroyed and the reference was still alive at such point)

[mgood7123]

one solution would be to forcefully collect all references before the arena is destroyed (regardless of weather such are reachable or not)

but i do not know how to do this

[mgood7123]

it seems we can do so via

static Res rootDestroy(Root root, void *p)
{
  /* we are inside a locked arena, we cannot */
  /* use 'mps_root_destroy' since it locks the arena */

  AVERT(Root, root);
  
  Arena arena;

  arena = RootArena(root);

  RootDestroy(root);

  return ResOK;
}

struct mps_thr_s;

// all mps ThreadStruct contain this layout

typedef struct mps_thr_s {      /* ANSI fake thread structure */
  Sig sig;                      /* design.mps.sig.field */
  Serial serial;                /* from arena->threadSerial */
  Arena arena;                  /* owning arena */
  RingStruct arenaRing;         /* attaches to arena */
  /* extra data per impl */
} ThreadStruct;

void destroy_roots(mps_arena_t mps_arena) {
  Arena arena = (Arena)mps_arena;

  /* lock the arena to prevent any roots from */
  /* being created while we are iterating them */

  ArenaEnter(arena);

  AVER(ArenaGlobals(arena)->clamped);          /* .assume.parked */
  AVER(arena->busyTraces == TraceSetEMPTY);    /* .assume.parked */

  Globals arenaGlobals = ArenaGlobals(arena);

  AVERT(Globals, arenaGlobals);

  RootsIterate(arenaGlobals, rootDestroy, 0);
  
  /* unregister all registered threads since */
  /* we have no roots associated with them */

  /*
  _`.req.register.multi`: It must be possible to register the same
  thread multiple times. (This is needed to support the situation where
  a program that does not use the MPS is calling into MPS-using code
  from multiple threads. On entry to the MPS-using code, the thread can
  be registered, but it may not be possible to ensure that the thread is
  deregistered on exit, because control may be transferred by some
  non-local mechanism such as an exception or ``longjmp()``. We don't
  want to insist that the client program keep a table of threads it has
  registered, because maintaining the table might require allocation,
  which might provoke a collection. See request.dylan.160252_.)


  *  Register/Deregister
  *
  *  Explicitly register/deregister a thread on the arena threadRing.
  *  Register returns a "Thread" value which needs to be used
  *  for deregistration.
  *
  *  Threads must not be multiply registered in the same arena.

  */

  Ring threadRing = ArenaThreadRing(arena);
  Ring node, next;
  AVERT(Ring, threadRing);
  RING_FOR(node, threadRing, next) {
    Thread thread = RING_ELT(Thread, arenaRing, node);

    /* we are inside a locked arena, we cannot */
    /* use 'mps_thread_unreg' since it locks the arena */

    AVER(ThreadCheckSimple(thread));
    
    AVER(arena == ThreadArena(thread)); /* is the thread arena always us? */

    ThreadDeregister(thread, arena);
  }

  ArenaLeave(arena);
}

and then collect the arena

    mps_arena_park(state->arena);        /* ensure no collection is running */

    printf("deinit - collect\n");

    destroy_roots(state->arena); // destroy all roots and unregister all threads from the given arena

    managed_obj_collect(state); // mps_arena_collect + managed_obj_mps_chat

    printf("deinit - collected\n");

    printf("deinit - shutdown\n");

    mps_ap_destroy(state->ap);       /* destroy ap before pool */
    mps_pool_destroy(state->pool);   /* destroy pool before fmt */
    mps_fmt_destroy(state->fmt);     /* destroy fmt before chain */
    mps_chain_destroy(state->chain); /* destroy chain before arena */
    mps_arena_destroy(state->arena);     /* last of all */

[mgood7123]

in a release build (with variant COOL) we get this

    ManagedObjState state;

    managed_obj_init(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE);
    
    managed_obj_t p1 = managed_obj_make(&state, malloc(500));
    managed_obj_t p2 = managed_obj_make(&state, malloc(500));

    p1 = (managed_obj_t) 0x6; // no longer referenced
    p2 = (managed_obj_t) 0x6; // no longer referenced

    managed_obj_deinit(&state);

[ RUN      ] MPS_collect2.1
reserved and comitted object 0x7f98a3580000 with pointer 0x6991c0
reserved and comitted object 0x7f98a3580018 with pointer 0x6993c0
deinit - collect
object 0x7f98a3600000 with pointer 0x6991c0 is being freed.
object 0x7f98a3600018 with pointer 0x6993c0 is being freed.
Total Collection duration: 0 seconds, 0 milliseconds, 327 microseconds
deinit - collected
deinit - shutdown
[       OK ] MPS_collect2.1 (1 ms)

[mgood7123]

do note

this is intended to forcefully collect all references held by an arena, with the intention of destroying the arena afterwards

provided no other arena also holds such references (not sure how to detect this)

this means that any references currently in use will become invalid (unless another arena also has a reference (not sure how to detect this))

(similar to freeing a memory arena and all pointers to such memory becoming UB (use-after-free) )

these should be detectable with debuggers and sanitizers

[rptb1]

Hello again! We are very interested in what you are doing, but our resource has been limited for the past few days. We will catch up.

I'll just point out that there is no communication between GCs in separate arenas. Each arena is like a separate world. Each client program (with its own object graph) is intended to have a single arena. The use case for multiple arenas would be if there were, e.g. two separate language runtimes sharing a process, both linking to the MPS as a library, and calling each other via some sort of FFI mechanism.

Pools within an arena are fully co-operative. The object graph can span pools and this will be handled correctly.

[mgood7123]

Hello again! We are very interested in what you are doing, but our resource has been limited for the past few days. We will catch up.

I'll just point out that there is no communication between GCs in separate arenas. Each arena is like a separate world. Each client program (with its own object graph) is intended to have a single arena. The use case for multiple arenas would be if there were, e.g. two separate language runtimes sharing a process, both linking to the MPS as a library, and calling each other via some sort of FFI mechanism.

Pools within an arena are fully co-operative. The object graph can span pools and this will be handled correctly.

so arena's cannot contain strong references to references allocated by other arena's ?

eg, a reference cannot be owned by multiple arena's ?

assuming this to be the case, if an arena is being destroyed, how might we go about migrating strong references from one arena to another arena (if needed) ?

[mgood7123]

do note

this is intended to forcefully collect all references held by an arena, with the intention of destroying the arena afterwards

provided no other arena also holds such references (not sure how to detect this)

this means that any references currently in use will become invalid (unless another arena also has a reference (not sure how to detect this))

(similar to freeing a memory arena and all pointers to such memory becoming UB (use-after-free) )

these should be detectable with debuggers and sanitizers

incedentally this MAY be an alternative to The only reliable way to ensure that all finalizable objects are finalized is to maintain a table of [weak references (1)](https://memory-pool-system.readthedocs.io/en/latest/glossary/w.html#term-weak-reference-1) to all such objects

[rptb1]

so arena's cannot contain strong references to references allocated by other arena's ?

Arenas can contain anything, but the MPS GC will not automatically find references (pointers) from one arena to another. There is nothing to stop you creating roots in one arena for another arena, just like you would create roots from any other memory. The arenas do not see each other: they are separate worlds.

eg, a reference cannot be owned by multiple arena's ?

Please explain what you mean by "owned" here.

assuming this to be the case, if an arena is being destroyed, how might we go about migrating strong references from one arena to another arena (if needed) ?

Please explain what you mean by "migrate" here. You can copy memory from one arena to another, e.g. with memcpy.

When dealing with GC finalization, it's very important to realise that there is no guarantee of promptness. See Cautions. Finalization by the GC is provided as an optimisation, and should not be used as a necessary part of an algorithm.

incedentally this MAY be an alternative to The only reliable way to ensure that all finalizable objects are finalized is to maintain a table of weak references (1) to all such objects

This is an interesting point. We don't currently guarantee that deleting all roots and collecting the world will recycle (and therefore finalize) every object, but we might be able to make that promise. We'll consider it.

Also, internally, finalization is managed by a "guardian pool" that has a list of all remaining finalizable objects. It may be possible to provide an interface to that list, so that a program can examine them all.

[rptb1]

this is intended to forcefully collect all references held by an arena, with the intention of destroying the arena afterwards

There may be some misunderstanding between us here. The MPS GC does not collect references (pointers to GC'd objects). It collects objects (blocks of memory). Objects can contain references, discovered by scanning.

[mgood7123]

eg, a reference cannot be owned by multiple arena's ?

Please explain what you mean by "owned" here.

never mind, i realized that arena's give out references, and they cannot "take ownership" of references

eg ref r; arena_own(&r, arena); // doesnt work that way

Please explain what you mean by "migrate" here

say we have ref_1 in arena_1, and we want to destroy arena_1 while still keeping ref_1 alive, by moving it from arena_1 to arena_2, how might we do this?

while this may contradict the following

they cannot "take ownership" of references

a reference could be transferred by taking ref_1 from a1, allocating a new ref ref_1_copy from a2, and manually copying the contents of ref_1 to ref_1_copy and then carefully destroying ref_1 such that upon finalization, any resources will not be deallocated as ref_1_copy now holds these resources

this might however, require all references to ref_1 be updated to point to ref_1_copy

There may be some misunderstanding between us here. The MPS GC does not collect references (pointers to GC'd objects). It collects objects (blocks of memory). Objects can contain references, discovered by scanning.

yea, assuming we have ref_40 and we set it to NULL, if upon collection it determines ref_40 cannot be located it will finalize (if registered) and collect it

by deleting the roots themselves we basically force ref_40 to be unlocatable even if it would have been locatable if the roots where still present

finalization is managed by a "guardian pool" that has a list of all remaining finalizable objects. It may be possible to provide an interface to that list, so that a program can examine them all.

true, however our goal is to explicitly finalize all unfinalized objects, eg turn all black objects to white objects, and keep all white objects as white objects such that everything in the arena is now white

[rptb1]

say we have ref_1 in arena_1, and we want to destroy arena_1 while still keeping ref_1 alive, by moving it from arena_1 to arena_2, how might we do this?

Are you saying ref_1 is a pointer inside an object allocated in arena_1?

The MPS will not examine references in arena_1 when it GCs arena_2. Arenas do not communicate. They are independent, separate worlds. So ref_1 is not ever keeping anything in arena_2 alive.

The only way that arena_2 will know about ref_1 (which is outside arena_2) is if you create a root in arena_2 that includes ref_1.

From your questions here, and in #260 , you seem to be focussed on the management of external resources (file handles, etc.), not on the management of memory blocks. Is that correct? Are you trying to use the MPS mainly to manage external resources automatically?

[rptb1]

I'll also just repeat that the MPS is designed to have one arena per program. The only use case for multiple arenas is if you have linked two independent programs (e.g. written by different organizations) into the same process, and they are sharing address space, but working independently.

[rptb1]

You are however welcome (and intended) to create multiple pools within your arena. That might be what you need. Let's say you create two instances of the AMC pool class, amcA and amcB. Now, if an object in amc1 (objA) is keeping an object in amc2 (objB) alive, and you destroy pool amc1, then objB will (eventually) be recycled (and possibly finalized). If you don't want that to happen, you need a pointer to objB in an object that is still alive, or in a root. How you do that is up to you, but the simplest case is that you just do objC.b = objB where objC is some other object, which is presumably not dead because you're talking about it.

[rptb1]

The reason I ask about external resources is that the MPS makes decisions about collection based on factors to do with space and time: how much memory is available, CPU time available, and so on. It is designed to manage memory in a way that the running program will not notice. It's very good at keeping pause times down for interactive applications. It does not think about external resources or make plans based on them. If you're trying to clean up external resources (e.g. file handles) promptly you're probably using the wrong tool. Finalization is only an optimization that the MPS offers in passing. You should not rely on finalization messages.

[mgood7123]

This is an interesting point. We don't currently guarantee that deleting all roots and collecting the world will recycle (and therefore finalize) every object, but we might be able to make that promise. We'll consider it.

additionally, since all roots are destroyed, reference resurrection should become effectively impossible

the only exception to this, is a root can be manually re-registered in the arena somehow, which requires a reference gaining access to the arena it belongs to, and then creating a root and then storing itself inside that root

[mgood7123]

to ensure parelization where possible

The MPS GC is a coroutine of your program and does not care about threads. You will not gain parallelization by creating arenas.

yes but to actually perform a collection, all threads registered to such arena (excluding the thead running the collection) must be suspended for the duration of the collection, right?

eg

GC

reg(GC); // register Thread 1 in GC.thread1 and GC.arena

reg(GC); // register Thread 2 in GC.thread2 and GC.arena

collect(GC); // Thread 3

run() // Thread 4

when collect is ran, the GC will suspend T1 and T2 but not T3 and T4 since it was not told about T3 and T4

this can in-turn be applied on a parallel basis

reg(GC_A); // register Thread 1 in GC_A.thread1 and GC_A.arena

reg(GC_B); // register Thread 2 in GC_B.thread1 and GC_B.arena

collect(GC_A); // Thread 3
collect(GC_B); // Thread 4

GC_A and GC_B can both collect in parallel without suspending eachother provided no gc objects are shared between their threads

[mgood7123]

HOWEVER such object is treated like a reference, as it is looked for in roots to determine if the object is still reachable right?

Please understand I'm using words pedantically so that we can understand each other. It's a tricky subject!

An object allocated by reserve/commit in a GC'd pool (such as one of class AMC) will be scanned to look for references inside it. The fields in that object are treated as references when they are passed to e.g. MPS_FIX by your format scanner.

So the references are the fields that your scan method fixes.

Those references are inside the object. We don't usually say that the object is a reference.

alright :)

[mgood7123]

HOWEVER such object is treated like a reference, as it is looked for in roots to determine if the object is still reachable right?

Please understand I'm using words pedantically so that we can understand each other. It's a tricky subject!

An object allocated by reserve/commit in a GC'd pool (such as one of class AMC) will be scanned to look for references inside it. The fields in that object are treated as references when they are passed to e.g. MPS_FIX by your format scanner.

So the references are the fields that your scan method fixes.

Those references are inside the object. We don't usually say that the object is a reference.

so in regards to references, ref_1->pointer.pointer itself is a field that we pass to FIX to tell it that that field may contain a reference, but ref_1->pointer.pointer itself is not scanned for references, eg ((FOO*)ref_1->pointer.pointer)->bar cannot be scanned to see if bar is a reference or not because we do not know its type at compile-time, and we cannot make it a root since the object itself may be moved via a moving garbage collector, invalidating the registered root

[mgood7123]

assuming so, in regards to topology #260 and also actually keeping references alive in other references

we would need some way to add objects to our fields at runtime in an effiecent manner, while also binding the fields to other object fields, eg a->field to b and a->ptr to b->ptr, right?

[rptb1]

yes but to actually perform a collection, all threads registered to such arena (excluding the thead running the collection) must be suspended for the duration of the collection, right?

No, not quite.

During some parts of a collection, all threads that might read or write any reference (usually all threads in your process) must be suspended, so that we can atomically update the graph of all objects in the process.

It's not safe to have some threads registered to one arena, and some to another arena, if there is any chance that the threads will look at or update any object, or share references with each other.

This is because the GC must act globally on the entire state of the program.

The details are very subtle. But your process and its threads are not stopped for the duration of the GC, only for short periods. The MPS is incremental and at quite a fine resolution. That is one reason it is successfully commercially: it does not interrupt the user even when there is a lot of GC to do.

[rptb1]

assuming so, in regards to topology #260 and also actually keeping references alive in other references

we would need some way to add objects to our fields at runtime in an effiecent manner, while also binding the fields to other object fields, eg a->field to b and a->ptr to b->ptr, right?

I'm out of time for today but I hope to help you again soon.

[mgood7123]

During some parts of a collection, all threads that might read or write any reference (usually all threads in your process) must be suspended, so that we can atomically update the graph of all objects in the process.

would it be possible to limit such to only registered threads?

[mgood7123]

assuming so, in regards to topology #260 and also actually keeping references alive in other references
we would need some way to add objects to our fields at runtime in an effiecent manner, while also binding the fields to other object fields, eg a->field to b and a->ptr to b->ptr, right?

I'm out of time for today but I hope to help you again soon.

alright

[rptb1]

would it be possible to limit such to only registered threads?

We must suspend all threads that can look at any object managed by the MPS. Basically because we are going to move them around without the program noticing.

[mgood7123]

would it be possible to limit such to only registered threads?

We must suspend all threads that can look at any object managed by the MPS. Basically because we are going to move them around without the program noticing.

so i assume we must explicitly register threads because we have no api to automatically register all existing/new threads?

[rptb1]

The amount of time that the threads are suspended is kept short. The MPS uses barriers so that it can GC and move objects without the threads noticing, even while they are running.

[mgood7123]

hmm ok

[rptb1]

so i assume we must explicitly register threads because we have no api to automatically register all existing/new threads?

We don't currently have any portable code to do this. We could possibly add some convenience code by looking at how, e.g. the BDWGC finds threads. It's much more "drop-in" than the MPS currently.

[mgood7123]

so

a GC registers threads as roots to scan

if a collect happens, ALL threads (both registered and unregistered) are suspended for short amounts of time

the more threads get registered, the more we need to scan

multiple pools can be used to seperate references, but all references in all pools must be scanned
likewise all registered thread stack/register roots must be scanned
likewise all registered and unregistered threads must be suspended for short amount during a GC collect

[mgood7123]

eg

/* ThreadRingSuspend -- suspend all threads on a ring, except the
 * current one.
 */

static Bool threadSuspend(Thread thread)
{
  Res res;
  pthread_t self;
  self = pthread_self();
  if (pthread_equal(self, thread->id)) /* .thread.id */
    return TRUE;

  /* .error.suspend: if PThreadextSuspend fails, we assume the thread
   * has been terminated. */
  AVER(thread->context == NULL);
  res = PThreadextSuspend(&thread->thrextStruct, &thread->context);
  AVER(res == ResOK);
  AVER(thread->context != NULL);
  /* design.thread-manager.sol.thread.term.attempt */
  return res == ResOK;
}

/* PThreadextSuspend -- suspend a thread
 *
 * <design/pthreadext#.impl.suspend>
 */

Res PThreadextSuspend(PThreadext target, MutatorContext *contextReturn)
{
  Ring node, next;
  Res res;
  int status;

  AVERT(PThreadext, target);
  AVER(contextReturn != NULL);
  AVER(target->context == NULL); /* multiple suspends illegal */

  /* Serialize access to suspend, makes life easier */
  status = pthread_mutex_lock(&pthreadextMut);
  AVER(status == 0);
  AVER(suspendingVictim == NULL);

  /* Threads are added to the suspended ring on suspension */
  /* If the same thread Id has already been suspended, then */
  /* don't signal the thread, just add the target onto the id ring */
  RING_FOR(node, &suspendedRing, next) {
    PThreadext alreadySusp = RING_ELT(PThreadext, threadRing, node);
    if (alreadySusp->id == target->id) {
      RingAppend(&alreadySusp->idRing, &target->idRing);
      target->context = alreadySusp->context;
      goto noteSuspended;
    }
  }

  /* Ok, we really need to suspend this thread. */
  suspendingVictim = target;
  status = pthread_kill(target->id, PTHREADEXT_SIGSUSPEND);
  if (status != 0) {
    res = ResFAIL;
    goto unlock;
  }

  /* Wait for the victim to acknowledge suspension. */
  while (sem_wait(&pthreadextSem) != 0) {
    if (errno != EINTR) {
      res = ResFAIL;
      goto unlock;
    }
  }

noteSuspended:
  AVER(target->context != NULL);
  RingAppend(&suspendedRing, &target->threadRing);
  *contextReturn = target->context;
  res = ResOK;

unlock:
  suspendingVictim = NULL;
  status = pthread_mutex_unlock(&pthreadextMut);
  AVER(status == 0);
  return res;
}

[rptb1]

a GC registers threads as roots to scan
if a collect happens, ALL threads (both registered and unregistered) are suspended for short amounts of time

Not quite.

The client program (your code) registers threads with the arena. This allows the GC to suspend them. It does not create roots. The MPS does not automatically find threads.

Only registered threads are suspended.

However, if an unregistered thread accesses any managed objects in the arena or any references to those objects from roots, you could get heap corruption. The corruption could occur much later and will be very difficult to debug. So we would recommend you register all threads. There are some circumstances (special debugging hacks usually) where you might want an unregistered thread.

Except in rare circumstances, you also want to create thread roots with your threads as well, so that they can freely reference the managed objects.

multiple pools can be used to seperate references, but all references in all pools must be scanned

You can separate your objects into pools for many reasons, e.g. different management policy, different parameters, easy mass destruction, etc.

The MPS needs to consider all references in the whole world, but it does not scan them every time. It uses remembered sets to optimise the scanning. A lot.

likewise all registered thread stack/register roots must be scanned

All thread roots are scanned, but basically yes.

likewise all registered and unregistered threads must be suspended for short amount during a GC collect

We don't suspend unregistered threads because we don't know about them. But I think you mean thread roots here.

[rptb1]

eg

You have found the relevant code :)

[mgood7123]

also

_`.sol.exclusive`: In order to meet `.req.exclusive`_, the arena
maintains a ring of threads (in ``arena->threadRing``) that have been
registered by the client program. When the MPS needs exclusive access
to memory, it suspends all the threads in the ring except for the
currently running thread. When the MPS no longer needs exclusive
access to memory, it resumes all threads in the ring.

_`.sol.exclusive.assumption`: This relies on the assumption that any
thread that might refer to, read from, or write to memory in
automatically managed pool classes is registered with the MPS. This is
documented in the manual under ``mps_thread_reg()``.

[mgood7123]

The client program (your code) registers threads with the arena. This allows the GC to suspend them. It does not create roots. The MPS does not automatically find threads.

Only registered threads are suspended.

However, if an unregistered thread accesses any managed objects in the arena or any references to those objects from roots, you could get heap corruption. The corruption could occur much later and will be very difficult to debug. So we would recommend you register all threads. There are some circumstances (special debugging hacks usually) where you might want an unregistered thread.

generally this would be the case

however if we can make certain assumptions about our managed objects, and assert that they hold true, then we can optimize for only select range of threads to be suspended

eg, threads that we can gaurentee will not touch our managed objects we do not require being suspended

[mgood7123]

multiple pools can be used to seperate references, but all references in all pools must be scanned

You can separate your objects into pools for many reasons, e.g. different management policy, different parameters, easy mass destruction, etc.

The MPS needs to consider all references in the whole world, but it does not scan them every time. It uses remembered sets to optimise the scanning. A lot.

alright

likewise all registered thread stack/register roots must be scanned

All thread roots are scanned, but basically yes.

likewise all registered and unregistered threads must be suspended for short amount during a GC collect

We don't suspend unregistered threads because we don't know about them. But I think you mean thread roots here.

so all registered threads must be suspended then (as previously stated, MPS cannot suspend a thread that it has not been registered with)

[mgood7123]

hmmm

i implemented a small scanner but my object does not seem to be collecting when it should

static mps_res_t managed_obj_scan(mps_ss_t ss, mps_addr_t base, mps_addr_t limit)
{
  mps_res_t res;
  MPS_SCAN_BEGIN(ss) {
    while (base < limit) {
      managed_obj_t obj = (managed_obj_t)base;
      switch (MANAGED_OBJECT_TYPE(obj)) {
      case MANAGED_OBJECT_TYPE_SCANNED_POINTER: {
        MPS_FIX_CALL(ss, res = obj->scanned_pointer.scanner(ss, obj->scanned_pointer.pointer));
        if (res != MPS_RES_OK)
            return res;
        base = (char *)base + MANAGED_OBJECT_ALIGN_OBJ(sizeof(managed_obj_scanned_pointer_s));
        break;
      }

#define MANAGED_OBJECT_FIX(ref) \
  do { \
    mps_addr_t _addr = (ref); /* copy to local to avoid type pun */ \
    mps_res_t res = MPS_FIX12(ss, &_addr); \
    if (res != MPS_RES_OK) return res; \
    (ref) = _addr; \
  } while(0)

TEST(MPS_finalize, 2_null) {
    ManagedObjState state;
    managed_obj_init(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE);

    volatile managed_obj_t p1 = managed_obj_make_with_finalizer(&state, new B_A(), +[](void*p){ delete static_cast<B_A*>(p); });
    volatile managed_obj_t p2 = managed_obj_make_scanned_with_finalizer(
        &state, new B(),
        // scanner
        +[](mps_ss_t ss, void*p) -> mps_res_t {
            MPS_SCAN_BEGIN(ss) {
                printf("b is %p, a is %p, fix b->a\n", p, ((B*)p)->a);
                MANAGED_OBJECT_FIX(((B*)p)->a);
            } MPS_SCAN_END(ss);
            return MPS_RES_OK;
        },
        // finalizer, set b->a to zero, b->a is currently a GC allocated object, so we must erase it
        +[](void*p){ printf("b is %p, a is %p, delete b\n", p, ((B*)p)->a); B * b = (B*)p; b->a = nullptr; delete b; printf("set b->a to nullptr, b is %p\n", b); }
    );

    // if we set 'a' to 'nullptr' then
    //   'B_A' (p1) gets finalized in the first  collect
    //   'B'   (p2) gets finalized in the second collect
    //
    // if we set 'a' to 'p1' then
    //   'B_A' (p1) gets finalized after all roots are destroyed
    //   'B'   (p2) gets finalized in the second collect
    //
    MANAGED_OBJECT_SCANNED_CAST(B, p2)->a = p1;

    p1 = nullptr;
    p2 = nullptr;

    printf("first collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("second collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("third collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    managed_obj_deinit(&state);
}

[ RUN      ] MPS_finalize.2_null
thread stack bottom: 0x7ffd2f937000
reserved and comitted object 0x7f0fac500000 with pointer 0x232c8b0
reserved and comitted object 0x7f0fac500020 with pointer 0x232c8d0

first collect, p may be collected

b is 0x232c8d0, a is 0x7f0fac500000, fix b->a

Total Collection duration: 0 seconds, 0 milliseconds, 365 microseconds

second collect, p may be collected

forward 0x7f0fac500020 to 0x7f0fac580000
b is 0x232c8d0, a is 0x7f0fac500000, fix b->a
pad 0x7f0fac500020
object 0x7f0fac580000 with pointer 0x232c8d0 is being finalized.
b is 0x232c8d0, a is 0x7f0fac500000, delete b
~B() (child is B_A)
set b->a to nullptr, b is 0x232c8d0
object 0x7f0fac580000 with pointer 0x232c8d0 has been freed, setting pointer to zero.

Total Collection duration: 0 seconds, 0 milliseconds, 472 microseconds

third collect, p may be collected

pad 0x7f0fac580020
pad 0x7f0fac500020

Total Collection duration: 0 seconds, 0 milliseconds, 482 microseconds

deinit - destroy all roots
deinit - destroyed all roots
deinit - collect

forward 0x7f0fac500000 to 0x7f0fac580000
pad 0x7f0fac500000
object 0x7f0fac580000 with pointer 0x232c8b0 is being finalized.
~B_A() (parent is A)
object 0x7f0fac580000 with pointer 0x232c8b0 has been freed, setting pointer to zero.

Total Collection duration: 0 seconds, 0 milliseconds, 688 microseconds

deinit - collected
deinit - shutdown
pad 0x7f0fac500040
pad 0x7f0fac580020
[       OK ] MPS_finalize.2_null (7 ms)

[mgood7123]

if i set p1 to a field and then fix it i get the same as b->a = p1

      case MANAGED_OBJECT_TYPE_SCANNED_POINTER: {
        MANAGED_OBJECT_FIX(obj->scanned_pointer.pointer);

    volatile managed_obj_t p1 = managed_obj_make_with_finalizer(&state, new B_A(), +[](void*p){ delete static_cast<B_A*>(p); });
    volatile managed_obj_t p2 = managed_obj_make_scanned_with_finalizer(
        &state, p1, // B_A (p1) is finalized after all roots have been destroyed
        // new B(),
        // scanner
        +[](mps_ss_t ss, void*p) -> mps_res_t {
        //     MPS_SCAN_BEGIN(ss) {
        //         // printf("b is %p, a is %p, fix b->a\n", p, ((B*)p)->a);
        //         MANAGED_OBJECT_FIX(p);
        //     } MPS_SCAN_END(ss);
            return MPS_RES_OK;
        },
        // finalizer
        +[](void*p) {}
    );

[mgood7123]

seems to be related to #259 (comment)

[mgood7123]

void level_2(ManagedObjState & state) {
    volatile managed_obj_t p1 = managed_obj_make_with_finalizer(&state, new B_A(), +[](void*p){ delete static_cast<B_A*>(p); });
    volatile managed_obj_t p2 = managed_obj_make_with_finalizer(&state, p1, +[](void*p) {} );

    managed_obj_assert(p2, MANAGED_OBJECT_TYPE_SCANNED_POINTER)->scanned_pointer.pointer = nullptr; // remove p1

    p1 = nullptr;
    p2 = nullptr;

    // p1 can be destroyed now, no references to it should exist

    printf("first collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("second collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("third collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p
}

void level_1(ManagedObjState & state) {
    level_2(state);
    printf("forth collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("fifth collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("sixth collect, p may be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p
}

TEST(MPS_finalize, 2_null) {
    ManagedObjState state;
    managed_obj_init(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE);

    level_1(state);

    // p1 should have been destroyed by now

    managed_obj_deinit(&state); // p1 is actually destroyed here
}

if we do this then our p1 (B_A) gets destroyed in the fifth collection while p2 gets destroyed in the first collection

[mgood7123]

like

    volatile managed_obj_t p2 = managed_obj_make_with_finalizer(&state, p1, +[](void*p) {} );

    managed_obj_assert(p2, MANAGED_OBJECT_TYPE_SCANNED_POINTER)->scanned_pointer.pointer = nullptr; // remove p1

should be the same as

    volatile managed_obj_t p2 = managed_obj_make_with_finalizer(&state, nullptr, +[](void*p) {} );

but for some reason it isnt

managed_obj_t managed_obj_make_with_finalizer(ManagedObjState * state, void * pointer, managed_obj_finalization_callback_t finalization_callback)
{
  return managed_obj_make_scanned_with_finalizer(state, pointer, managed_obj_no_scan_callback, finalization_callback);
}

managed_obj_t managed_obj_make_scanned_with_finalizer(ManagedObjState * state, void * pointer, managed_obj_scanned_pointer_scan_fn_t scanner, managed_obj_finalization_callback_t finalization_callback)
{
  managed_obj_t obj;
  size_t size = MANAGED_OBJECT_ALIGN_OBJ(sizeof(managed_obj_scanned_pointer_s));
  while(1) {
    mps_res_t res = mps_reserve((mps_addr_t*)&obj, state->ap, size);
    if (res != MPS_RES_OK) managed_obj_error("out of memory in make_pointer");
    obj->scanned_pointer.type = MANAGED_OBJECT_TYPE_SCANNED_POINTER;
    obj->scanned_pointer.pointer = pointer;
    obj->scanned_pointer.scanner = scanner;
    obj->scanned_pointer.finalization_callback = finalization_callback;
    if (mps_commit(state->ap, obj, size)) {
      break;
    }
  }
  printf("reserved and comitted object %p with pointer %p\n", obj, obj->scanned_pointer.pointer);

  state->total += sizeof(managed_obj_scanned_pointer_s);

  mps_finalize(state->arena, (mps_addr_t*)&obj);

  return obj;
}

even tho in both obj->scanned_pointer.pointer ends up being set to nullptr

[mgood7123]

using stack frames seems to work, as even if the compiler attaches the variable to the frame/register they will be gone when the frame exits

l_1 < outside func(), p1 and p2 do not exist here, they cannot be found in stack/registers
  func();
    l_2 < inside func(), p1 and p2 exist here, in stack/registers
  ... < func() return, p1 and p2 do not exist here, they cannot be found in stack/registers

[mgood7123]

also using custom memory works

TEST(MPS_finalize, 2_null_user_mem_6) {
    ManagedObjState state;

    managed_obj_t memory[mem_count];

    managed_obj_init_with_user_memory(&state, MANAGED_OBJECT_DEFAULT_ARENA_SIZE, memory, sizeof(void*)*mem_count);

    memory[0] = managed_obj_make_with_finalizer(&state, new B_A(), +[](void*p){ delete static_cast<B_A*>(p); });
    memory[1] = managed_obj_make_scanned_with_finalizer(
        &state, new B(),
        // scanner
        //
        // if we do not scan 'a' then 'memory[0]' will be collected
        //   when 'memory[0]' is set to 0x0
        //
        +[](mps_ss_t ss, void*p) -> mps_res_t {
            MPS_SCAN_BEGIN(ss) {
                MANAGED_OBJECT_FIX(static_cast<B*>(p)->a);
            } MPS_SCAN_END(ss);
            return MPS_RES_OK;
        },
        // finalizer
        +[](void*p){ MANAGED_OBJECT_ZERO_AND_DELETE(B, p); }
    );
    MANAGED_OBJECT_SCANNED_CAST(B, memory[1])->a = memory[0];

    printf("first collect, p should not be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("second collect, p should not be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("third collect, p should not be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    memory[0] = nullptr;

    printf("forth collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("fifth collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("sixth collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    memory[1] = nullptr;

    printf("seventh collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("eigth collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    printf("ninth collect, p should be collected\n");
    managed_obj_collect(&state); // should collect the original pointer assigned to p

    // p1 should have been destroyed by now

    managed_obj_deinit(&state); // p1 is actually destroyed here
}

ignore the 'p should be collected' stuff, im not sure how to verify collection when using stack memory instead of user memory

[mgood7123]

imma close this for now