Skip to content

Commit

Permalink
Add Mmap allocator information to error message when failing with MMa…
Browse files Browse the repository at this point in the history
…p failed (facebookincubator#9455)

Summary:
We currently have a lack of information on the current status of the MMap allocator when we fail in the logging. Here is an example of what we see.

```
E20240410 00:12:53.207183 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.218799 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.228353 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.239408 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.250753 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.259156 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.266788 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.274092 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.287583 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.296489 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
I20240410 00:12:53.296515 21387 AsyncDataCache.cpp:799] [CACHE] Backoff in allocation contention for 23.67ms
E20240410 00:12:53.327771 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.342656 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.387915 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.413861 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.463887 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.501030 21387 MmapAllocator.cpp:344] [MEM] Mmap failed with 4194304 pages use MmapArena false
E20240410 00:12:53.521816 21387 Exceptions.h:69] Line: /app/presto-native-execution/velox/velox/common/memory/MemoryPool.cpp:1198, Function:handleAllocationFailure, Expression:  allocateContiguous failed with 4194304 pages from Memory Pool[op.2149.4.63.HashBuild LEAF root[20240410_001004_00013_5sjzc_123] parent[node.2149] MMAP track-usage thread-safe]<max capacity 437.00GB unlimited capacity used 0B available 0B reservation [used 0B, reserved 0B, min 0B] counters [allocs 1, frees 0, reserves 0, releases 1, collisions 0])> Mmap failed with 4194304 pages use MmapArena false Failed to evict from cache state: AsyncDataCache:
Cache size: 57.10GB tinySize: 228.15KB large size: 57.09GB
Cache entries: 15536 read pins: 128 write pins: 0 pinned shared: 126.07MB pinned exclusive: 0B
 num write wait: 19610 empty entries: 112530
Cache access miss: 235115 hit: 974160 hit bytes: 2.51TB eviction: 219579 eviction checks: 296298 aged out: 0
Prefetch entries: 2 bytes: 890.92KB
Alloc Megaclocks 610291
Allocated pages: 32586619 cached pages: 14967066
, Source: RUNTIME, ErrorCode: MEM_ALLOC_ERROR
```

By adding toString(), we will be able to see output like
```
Memory Allocator[MMAP total capacity 461.00GB free capacity 66.99GB allocated pages 103287293 mapped pages 104333972 external mapped pages 7670401
[size 1: 33382(130MB) allocated 60831 mapped]
[size 2: 43805(342MB) allocated 343492 mapped]
[size 4: 34963(546MB) allocated 47499 mapped]
[size 8: 28204(881MB) allocated 38870 mapped]
[size 16: 35949(2246MB) allocated 47261 mapped]
[size 32: 47909(5988MB) allocated 51124 mapped]
[size 64: 36067(9016MB) allocated 36067 mapped]
[size 128: 45642(22821MB) allocated 45642 mapped]
[size 256: 331530(331530MB) allocated 331532 mapped]
]
```

Pull Request resolved: facebookincubator#9455

Reviewed By: amitkdutta

Differential Revision: D56230522

Pulled By: xiaoxmeng

fbshipit-source-id: f56832115468c0354680381c27f5f175580184ac
  • Loading branch information
mohsaka authored and Joe-Abraham committed Jun 7, 2024
1 parent 578c478 commit 51d11e9
Showing 1 changed file with 3 additions and 2 deletions.
5 changes: 3 additions & 2 deletions velox/common/memory/MmapAllocator.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -340,10 +340,11 @@ bool MmapAllocator::allocateContiguousImpl(
}
if (data == nullptr || data == MAP_FAILED) {
const std::string errorMsg = fmt::format(
"Mmap failed with {} pages use MmapArena {}, errno {}",
"Mmap failed with {} pages use MmapArena {}, errno {}, Mmap Allocator: {}",
numPages,
(useMmapArena_ ? "true" : "false"),
folly::errnoStr(errno));
folly::errnoStr(errno),
toString());
VELOX_MEM_LOG(ERROR) << errorMsg;
setAllocatorFailureMessage(errorMsg);
// If the mmap failed, we have unmapped former 'allocation' and the extra to
Expand Down

0 comments on commit 51d11e9

Please sign in to comment.