[v1.15.x] util/memhooks: Fix a bug when calculating mprotect region

[shijin-aws]

backport #9552

[j-xiong]

@shijin-aws Could you rebase this one to pick up CI changes in the v1.15.x branch? ubuntu 18.04 and macos 10.5 runners don't seem to be working any more.

[shijin-aws]

macos check failed

common/shared.c:3677:45: error: use of undeclared identifier 'mask'
        return sched_setaffinity(0, sizeof(mask), &mask);
                                                   ^
9 errors generated.
make[1]: *** [common/shared.lo] Error 1
make[1]: *** Waiting for unfinished jobs....
common/hmem.c:172:25: error: no member named '__builtin___memset_chk' in 'struct ft_hmem_ops'
        return hmem_ops[iface].memset(device, buf, value, size);
               ~~~~~~~~~~~~~~~ ^
/Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/secure/_string.h:77:3: note: expanded from macro 'memset'
                __builtin___memset_chk (dest, __VA_ARGS__, __darwin_obsz0 (dest))
                ^
common/hmem.c:172:32: warning: incompatible integer to pointer conversion passing 'uint64_t' (aka 'unsigned long long') to parameter of type 'const void *' [-Wint-conversion]
        return hmem_ops[iface].memset(device, buf, value, size);
                                      ^~~~~~
/Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/secure/_string.h:77:62: note: expanded from macro 'memset'
                __builtin___memset_chk (dest, __VA_ARGS__, __darwin_obsz0 (dest))
                                                                           ^~~~
/Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/secure/_common.h:38:55: note: expanded from macro '__darwin_obsz0'
#define __darwin_obsz0(object) __builtin_object_size (object, 0)
                                                      ^~~~~~
1 warning and 1 error generated.
make[1]: *** [common/hmem.lo] Error 1
make: *** [all] Error 2
Error: Process completed with exit code 2.

Doesn't relate to this change though

[j-xiong]

@shijin-aws That's right. Some CPU affinity setting related definitions are missing for osx fabtests build. It's also weird that the compiler would replace the struct member memset with a built-in name.

[j-xiong]

I would ignore the macos build result for this PR.

[shijin-aws]

Same segfaults as v1.17.x backport, will look into

[j-xiong]

Please rebase to allow build checks to pass and get Intel CI to run (instead of being stuck). Same for #9557 and #9558.

[a-szegel]

bot:aws:retest