iocp: fix crash, GetQueuedCompletionStatus() write freed WSAOVERLAPPED memory #4136

jimying · 2024-11-07T06:34:11Z

Try to fix issue #985. The idea is to call CancelIoEx() for the unregistering socket/key to cancel all pending operations of the key. However, as CancelIoEx() is basically asynchronous, this also makes the key unregistration asynchronous, so here are some consequences:

The memory for operation key must not be freed until the operation cancellation is completed, so instead of using app supplied operation key, the IOCP will use an internal copy.
Key must always have a group lock (can be supplied by app), which will handle key's resources release.

nanangizz · 2024-11-07T08:55:30Z

Great! I assume you've tested this and it does fix the issue :)

I think we also need to run under somekind of stress test, e.g: using ioqueue test in pjlib-test, to make sure all memory pools (of pending ops) are properly released. Note that the after an ioqueue key is unregistered, the key will be put into the closing-key-list and soon into the free-key-list to be reused by another socket. We need to make sure that all pending op has been freed before the key is freed & reused.

Next, perhaps we can apply a little bit optimization, e.g: instead of mem-pool for each pending-op, perhaps mem-pool per ioqueue-key to avoid multiple alloc+free for multiple pending-op, using same mechanism as ioqueue key (employing additional list for keeping unused pending-op instances to be reused later).

jimying · 2024-11-07T09:10:49Z

Note:
this only fix the issue when PJ_IOQUEUE_HAS_SAFE_UNREG=1;
PJ_IOQUEUE_HAS_SAFE_UNREG=0 still has memory error

nanangizz · 2024-11-11T02:55:28Z

Note: this only fix the issue when PJ_IOQUEUE_HAS_SAFE_UNREG=1; PJ_IOQUEUE_HAS_SAFE_UNREG=0 still has memory error

When PJ_IOQUEUE_HAS_SAFE_UNREG==1, the key won't be reused for 500ms (configurable via PJ_IOQUEUE_KEY_FREE_DELAY), this is a bit risky actually, e.g: on high CPU load, the cancellation may take longer?

nanangizz · 2024-11-14T07:15:11Z

Tried to run pjlib-test with this PR patch on VS2005 and pool debugging enabled (so pool uses normal malloc/free(), by setting PJ_POOL_DEBUG to 1 in config_site.h), I got an assertion:

 	_wassert(const wchar_t * expr=0x004d1e90, const wchar_t * filename=0x004d1cf0, unsigned int lineno=588) Line 212	C
 	pj_ioqueue_register_sock2(pj_pool_t * pool=0x023a29b8, pj_ioqueue_t * ioqueue=0x023a2a2c, long sock=388, pj_grp_lock_t * grp_lock=0x00000000, void * user_data=0x00000002, const pj_ioqueue_callback * cb=0x0019fb54, pj_ioqueue_key_t * * key=0x0019fbec) Line 588 + 0x2c bytes	C
 	pj_ioqueue_register_sock(pj_pool_t * pool=0x023a29b8, pj_ioqueue_t * ioqueue=0x023a2a2c, long sock=388, void * user_data=0x00000002, const pj_ioqueue_callback * cb=0x0019fb54, pj_ioqueue_key_t * * key=0x0019fbec) Line 665 + 0x1f bytes	C
 	unregister_test(const pj_ioqueue_cfg * cfg=0x0019fd34) Line 554 + 0x23 bytes	C
 	udp_ioqueue_test_imp(const pj_ioqueue_cfg * cfg=0x0019fd34) Line 1190 + 0x9 bytes	C
 	udp_ioqueue_test() Line 1255 + 0x9 bytes	C
 	test_inner() Line 171 + 0x2a bytes	C
 	test_main() Line 245 + 0x5 bytes	C

Not sure if this is the same issue, but this assertion does not happen when using ioqueue select.

jimying · 2024-11-14T07:50:32Z

@nanangizz no this patch, Is there this assert?

nanangizz · 2024-11-14T07:56:41Z

@nanangizz no this patch, Is there this assert?

Yes, same assert without this patch.

pjlib/src/pj/ioqueue_winnt.c

jimying · 2024-11-17T01:37:50Z

Tried to run pjlib-test with this PR patch on VS2005 and pool debugging enabled (so pool uses normal malloc/free(), by setting PJ_POOL_DEBUG to 1 in config_site.h), I got an assertion:

 	_wassert(const wchar_t * expr=0x004d1e90, const wchar_t * filename=0x004d1cf0, unsigned int lineno=588) Line 212	C
 	pj_ioqueue_register_sock2(pj_pool_t * pool=0x023a29b8, pj_ioqueue_t * ioqueue=0x023a2a2c, long sock=388, pj_grp_lock_t * grp_lock=0x00000000, void * user_data=0x00000002, const pj_ioqueue_callback * cb=0x0019fb54, pj_ioqueue_key_t * * key=0x0019fbec) Line 588 + 0x2c bytes	C
 	pj_ioqueue_register_sock(pj_pool_t * pool=0x023a29b8, pj_ioqueue_t * ioqueue=0x023a2a2c, long sock=388, void * user_data=0x00000002, const pj_ioqueue_callback * cb=0x0019fb54, pj_ioqueue_key_t * * key=0x0019fbec) Line 665 + 0x1f bytes	C
 	unregister_test(const pj_ioqueue_cfg * cfg=0x0019fd34) Line 554 + 0x23 bytes	C
 	udp_ioqueue_test_imp(const pj_ioqueue_cfg * cfg=0x0019fd34) Line 1190 + 0x9 bytes	C
 	udp_ioqueue_test() Line 1255 + 0x9 bytes	C
 	test_inner() Line 171 + 0x2a bytes	C
 	test_main() Line 245 + 0x5 bytes	C

Not sure if this is the same issue, but this assertion does not happen when using ioqueue select.

I found the reason: key double unregister.
Fixed: when key->closing = 1, not unregister.
now test passed.

nanangizz · 2024-11-22T06:49:18Z

Thanks @jimying .

Honestly I haven't got a chance to reproduce the original issue and test the proposed solution. I believe you are using this ioqueue in real world, experienced the issue, and find this solution does work, is that correct?

Next, here are few notes about the proposed solution:

As you've mentioned, the approach requires PJ_IOQUEUE_HAS_SAFE_UNREG to work, so I think this mode has to be enforced somehow.
Also as I've mentioned before, the safe key unregistration relies on timer (default is 500ms), there is a risk when CPU load is high, increasing the timeout is not ideal as the risk itself is actually still there, only reduced to some degree. Perhaps the unregistration should rely on zero-pending-operation instead, or combination of them somehow.
Memory pool per operation can be optimized. Note that a single ioqueue key can have many & rapid operations, e.g: receiving RTP packets, so creating & releasing pool for each operation does not sound very optimized. An idea is to have some pool of pending_op (just like pool of key), may be owned by ioqueue or by key.

Also, this ioqueue has been disabled for quite sometime and some improvement in the ioqueue area may not be integrated into this ioqueue, e.g: group lock for key. So please understand that there may still be some steps required to enable this ioqueue again :)

jimying · 2024-11-23T15:59:19Z

@nanangizz i write a simple demo to reproduce the crash issue in msys2, #4172

I have tested it, in old code, it can 100% reproduce the crash.

To test new code we can git cherry-pick the demo patch to this branch.

nanangizz · 2024-11-26T07:12:34Z

Thanks @jimying.

…D memory try to fix issue pjsip#985

… to free list

sauwming · 2024-12-13T05:34:06Z

Hi @jimying, please let us know whether you plan to incorporate @nanangizz suggestions above.
In my opinion, the first two (enforcingPJ_IOQUEUE_HAS_SAFE_UNREG and zero pending-op) are important and shouldn't require much modification.
While for the last suggestion (memory pool), I think it's optional if you don't have the time necessary to implement it. Just put a note in the code for future optimisation.

jimying · 2024-12-13T06:00:32Z

@sauwming sorry, I will submit it as soon as possible today or tomorrow

jimying · 2024-12-13T07:40:05Z

enforcingPJ_IOQUEUE_HAS_SAFE_UNREG and zero pending-op

new commits do:

remove macro PJ_IOQUEUE_HAS_SAFE_UNREG, only keep PJ_IOQUEUE_HAS_SAFE_UNREG=1 logic
remove closing_list, when ref_count=0 return key to freelist.
In order to achieve，add ref when alloc pending-op and dec ref when free it

pjlib/src/pj/ioqueue_winnt.c

The pool is owned by key/socket, instead of by ioqueue, to avoid possible infinite memory grow in ioqueue.

Update ioq_stress_test not to use the global group lock for key registration, as otherwise the keys won't be released until the global group lock is destroyed (i.e: after ioqueue destroy).

nanangizz · 2025-01-24T07:55:39Z

I think this is ready for review @jimying , @sauwming , @trengginas.

pjlib/src/pj/ioqueue_winnt.c

pjlib/src/pjlib-test/ioq_iocp_unreg_test.c

pjlib/src/pjlib-test/ioq_stress_test.c

- added info to clarify codes - added copyright for test code - minors.

nanangizz · 2025-01-27T04:30:58Z

The last commit should cover all review comments above.

@jimying, re: copyright text, feel free to change the name :)
Btw, as mentioned in the top of the file, the test can repro the issue on Win 10 & MSVC2005, but cannot on Windows 11 & MSVC 2022. Also the test will be run for all ioqueue types (originally for iocp only), just in case.

sauwming · 2025-01-27T04:36:11Z

pjlib/src/pjlib-test/ioq_iocp_unreg_test.c

@@ -0,0 +1,208 @@
+/*
+ * Copyright (C) 2024 jimying at github dot com.
+ * Copyright (C) 2024 Teluu Inc. (http://www.teluu.com)


If I'm not mistaken, Teluu is usually put above, for two reasons: 1. to signify that the original author has agreed to contribute it to Teluu (as per CLA), 2. to make it easier to update copyright year (i.e. only the latest/first copyright info will get updated, the rest will remain the same).

jimying · 2025-01-27T07:50:58Z

pjlib/src/pj/ioqueue_winnt.c

+ * operations must be cancelled. As cancelling ops is asynchronous,
+ * IOCP destroy may need to wait for the maximum time specified here.
+ */
+#define TIMEOUT_CANCEL_OP   5000


TIMEOUT_CANCEL_OP macro is unused. WAIT_KEY_MS (in pj_ioqueue_destroy()) the same value?

Actually WAIT_KEY_MS should have been replaced by TIMEOUT_CANCEL_OP, so WAIT_KEY_MS is unused (and undefined).

jimying · 2025-01-27T07:54:47Z

pjlib/src/pj/ioqueue_winnt.c


        pj_list_push_back(&ioqueue->free_list, key);
    }
-#endif
+    ioqueue->max_fd = pj_list_size(&ioqueue->free_list); // max_fd;


why compute again use pj_list_size()? better revert to ioqueue->max_fd= max_fd

oops right, forgot to revert.

jimying · 2025-01-27T07:59:34Z

pjlib/src/pj/ioqueue_winnt.c

+        pj_gettickcount(&timeout);
+        if (PJ_TIME_VAL_GTE(timeout, stop)) {
+            PJ_LOG(3, (THIS_FILE, "Warning, IOCP destroy timeout in waiting "
+                       "for cancelling ops, after %dms, pending keys=%d",


minor build warning: format '%d' expects argument of type 'int', but argument 4 has type 'pj_size_t'

jimying mentioned this pull request Nov 7, 2024

Write to freed memory area with IoCompletionPort (IOCP) ioqueue backend may lead to application crash #985

Closed

nanangizz reviewed Nov 14, 2024

View reviewed changes

pjlib/src/pj/ioqueue_winnt.c Outdated Show resolved Hide resolved

jimying marked this pull request as ready for review November 22, 2024 02:52

nanangizz requested review from trengginas and sauwming November 22, 2024 06:49

nanangizz added type: enhancement type: bug component: pjlib labels Nov 22, 2024

nanangizz added this to the release-2.15 milestone Nov 22, 2024

sauwming mentioned this pull request Nov 22, 2024

stack implementation and tests #4116

Closed

nanangizz modified the milestones: release-2.15, release-2.16 Dec 3, 2024

jimying added 7 commits December 3, 2024 20:55

iocp: fix crash, GetQueuedCompletionStatus() write freed WSAOVERLAPPE…

ad71e9d

…D memory try to fix issue pjsip#985

wrap some logic to functions to make code clean

ccecf7e

remove one useless code

7908954

Simplify the cancel pending io logic

0fc931d

avoid double key-unregistration

ea00645

minor optimization: no memcpy

9a5cbb7

Add check to confirm there're no pending operations before key return…

99adb39

… to free list

jimying force-pushed the iocp-mem branch from cfe33e9 to 99adb39 Compare December 3, 2024 12:57

jimying added 2 commits December 13, 2024 15:24

only keep the PJ_IOQUEUE_HAS_SAFE_UNREG=1 logic

be88a99

remove closing_list, use ref_count=0 to release key

ab10f5e

remove useless free_time

e53fadd

sauwming reviewed Dec 20, 2024

View reviewed changes

pjlib/src/pj/ioqueue_winnt.c Outdated Show resolved Hide resolved

nanangizz added 15 commits January 15, 2025 10:05

Merge branch 'master' into pr/4136

c52406b

Use pending-op pool

b7a8338

The pool is owned by key/socket, instead of by ioqueue, to avoid possible infinite memory grow in ioqueue.

Integrate group lock

9ea705a

Update ioq_stress_test not to use the global group lock for key registration, as otherwise the keys won't be released until the global group lock is destroyed (i.e: after ioqueue destroy).

Merge branch 'master' into pr/4136

8edf128

Update ci-win.yml

4c287c9

Update ci-win.yml

f39b7ae

Update ci-win.yml

e517f94

Update ci-win.yml

fea5fef

Update ci-win.yml

974dc46

Update ci-win.yml

62c1763

Merge branch 'master' into pr/4136

cca809c

add the reproducing test, also fix minor bugs

c633102

Update Makefile

d3fddb3

Merge branch 'master' into pr/4136

6c9599c

Update ci-win.yml

3823a04

sauwming reviewed Jan 27, 2025

View reviewed changes

Changes based on comments

9e0d2cb

- added info to clarify codes - added copyright for test code - minors.

sauwming approved these changes Jan 27, 2025

View reviewed changes

update copyright line order

c26067d

jimying commented Jan 27, 2025

View reviewed changes

fixes based on comments

c07ee83

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

iocp: fix crash, GetQueuedCompletionStatus() write freed WSAOVERLAPPED memory #4136

iocp: fix crash, GetQueuedCompletionStatus() write freed WSAOVERLAPPED memory #4136

jimying commented Nov 7, 2024 •

edited by nanangizz

Loading

nanangizz commented Nov 7, 2024

jimying commented Nov 7, 2024

nanangizz commented Nov 11, 2024

nanangizz commented Nov 14, 2024 •

edited

Loading

jimying commented Nov 14, 2024

nanangizz commented Nov 14, 2024

jimying commented Nov 17, 2024 •

edited

Loading

nanangizz commented Nov 22, 2024

jimying commented Nov 23, 2024 •

edited

Loading

nanangizz commented Nov 26, 2024

sauwming commented Dec 13, 2024

jimying commented Dec 13, 2024

jimying commented Dec 13, 2024 •

edited

Loading

nanangizz commented Jan 24, 2025

nanangizz commented Jan 27, 2025

sauwming Jan 27, 2025

jimying Jan 27, 2025

nanangizz Jan 27, 2025

jimying Jan 27, 2025

nanangizz Jan 27, 2025

jimying Jan 27, 2025

nanangizz Jan 27, 2025

iocp: fix crash, GetQueuedCompletionStatus() write freed WSAOVERLAPPED memory #4136

Are you sure you want to change the base?

iocp: fix crash, GetQueuedCompletionStatus() write freed WSAOVERLAPPED memory #4136

Conversation

jimying commented Nov 7, 2024 • edited by nanangizz Loading

nanangizz commented Nov 7, 2024

jimying commented Nov 7, 2024

nanangizz commented Nov 11, 2024

nanangizz commented Nov 14, 2024 • edited Loading

jimying commented Nov 14, 2024

nanangizz commented Nov 14, 2024

jimying commented Nov 17, 2024 • edited Loading

nanangizz commented Nov 22, 2024

jimying commented Nov 23, 2024 • edited Loading

nanangizz commented Nov 26, 2024

sauwming commented Dec 13, 2024

jimying commented Dec 13, 2024

jimying commented Dec 13, 2024 • edited Loading

nanangizz commented Jan 24, 2025

nanangizz commented Jan 27, 2025

sauwming Jan 27, 2025

Choose a reason for hiding this comment

jimying Jan 27, 2025

Choose a reason for hiding this comment

nanangizz Jan 27, 2025

Choose a reason for hiding this comment

jimying Jan 27, 2025

Choose a reason for hiding this comment

nanangizz Jan 27, 2025

Choose a reason for hiding this comment

jimying Jan 27, 2025

Choose a reason for hiding this comment

nanangizz Jan 27, 2025

Choose a reason for hiding this comment

jimying commented Nov 7, 2024 •

edited by nanangizz

Loading

nanangizz commented Nov 14, 2024 •

edited

Loading

jimying commented Nov 17, 2024 •

edited

Loading

jimying commented Nov 23, 2024 •

edited

Loading

jimying commented Dec 13, 2024 •

edited

Loading