[BUG] Out-of-memory issue with limited memory VM

[jnv]

Originally reported by @StorytellerCZ.

Expected Behavior

When deploying the Meteor application with OneSDK installed, one expects it to work without any issues on standard hosting.

Current Behavior

After deploying the application to Meteor Galaxy (a Meteor-specific cloud hosting solution) the application immediately (?) crashes with OOM error:

https://gist.github.com/StorytellerCZ/4ce7196f416972ec2760d384def74439

Steps to Reproduce

To be added. I will try to reproduce this with minimal application; it's possible we are reaching Galaxy limit in addition to other libraries.

Your Environment

• Version used: @superfaceai/[email protected]
• Environment name and version: Node 14.18.2
• Operating System and version: Ubuntu 20.04 (see docs)

[jnv]

I think this isn't specific to Meteor per se, but rather by memory limitations imposed by the hosting, so for starters we could try running just OneSDK with higher GC pressure and make sure it doesn't leak.

In general it would be helpful to learn what is the overhead of Comlink interpreter regarding memory usage, perhaps there are some possibilities of easy optimizations.

[StorytellerCZ]

To be more specific it runs if you get 1GB container. It goes to 100% CPU during start (tested with v1.0.1):

On smaller containers it crashes:

4dy23
2021-12-01 12:46:58+01:00<--- Last few GCs --->
4dy23
2021-12-01 12:46:58+01:00
4dy23
2021-12-01 12:46:58+01:00[7:0x621d600] 132997 ms: Scavenge (reduce) 253.0 (255.8) -> 252.3 (256.8) MB, 2.6 / 0.0 ms (average mu = 0.478, current mu = 0.219) allocation failure
4dy23
2021-12-01 12:46:58+01:00[7:0x621d600] 133797 ms: Scavenge (reduce) 253.8 (256.5) -> 252.6 (257.3) MB, 4.1 / 0.0 ms (average mu = 0.478, current mu = 0.219) allocation failure
4dy23
2021-12-01 12:46:58+01:00[7:0x621d600] 134394 ms: Scavenge (reduce) 253.3 (256.3) -> 252.9 (257.5) MB, 101.6 / 0.0 ms (average mu = 0.478, current mu = 0.219) allocation failure
4dy23
2021-12-01 12:46:58+01:00
4dy23
2021-12-01 12:46:58+01:00
4dy23
2021-12-01 12:46:58+01:00<--- JS stacktrace --->
4dy23
2021-12-01 12:46:58+01:00
4dy23
2021-12-01 12:46:58+01:00FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
4dy23
2021-12-01 12:46:58+01:00 1: 0xa389b0 node::Abort() [node]
4dy23
2021-12-01 12:46:58+01:00 2: 0x96e0af node::FatalError(char const*, char const*) [node]
4dy23
2021-12-01 12:46:58+01:00 3: 0xbb7a4e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
4dy23
2021-12-01 12:46:58+01:00 4: 0xbb7dc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
4dy23
2021-12-01 12:46:58+01:00 5: 0xd73fd5 [node]
4dy23
2021-12-01 12:46:58+01:00 6: 0xda496e v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [node]
4dy23
2021-12-01 12:46:58+01:00 7: 0xdb09a6 v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [node]
4dy23
2021-12-01 12:46:58+01:00 8: 0xd9cb3f v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [node]
4dy23
2021-12-01 12:46:58+01:00 9: 0xd9cdb8 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [node]
4dy23
2021-12-01 12:46:58+01:0010: 0xd8f699 v8::internal::ItemParallelJob::Run() [node]
4dy23
2021-12-01 12:46:58+01:0011: 0xdb2900 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [node]
4dy23
2021-12-01 12:46:58+01:0012: 0xdb319c v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [node]
4dy23
2021-12-01 12:46:58+01:0013: 0xdb3365 v8::internal::MarkCompactCollector::Evacuate() [node]
4dy23
2021-12-01 12:46:58+01:0014: 0xdc5361 v8::internal::MarkCompactCollector::CollectGarbage() [node]
4dy23
2021-12-01 12:46:58+01:0015: 0xd81628 v8::internal::Heap::MarkCompact() [node]
4dy23
2021-12-01 12:46:58+01:0016: 0xd83118 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
4dy23
2021-12-01 12:46:58+01:0017: 0xd8655c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
4dy23
2021-12-01 12:46:58+01:0018: 0xd54c3b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
4dy23
2021-12-01 12:46:58+01:0019: 0x109d21f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
4dy23
2021-12-01 12:46:58+01:0020: 0x1446379 [node]
4dy23
2021-12-01 12:47:02+01:00Application exited with signal: aborted
4dy23
2021-12-01 12:47:37+01:00The container has crashed. A new container will be started to replace it.

[jnv]

@StorytellerCZ Thanks, I will add info about the Node version. Could you please add what's the version of @superfaceai/one-sdk you use?

I'd assume the high CPU usage would be quite normal for most projects upon the initial start, or is it not the case when you omit OneSDK?

Also to be sure, the application crashes upon the start and not upon profile perform, right? So either we are allocating something huge on SDK initialization, or we just have too much dependencies.

[StorytellerCZ]

I'll try to get a repro on minimum Meteor project. It would be best if we could get it running with free hosting option or at least the tiny container. For the latest report I have used the latest 1.0.1 version.
It is as you said, the problem occurs on container initialization.

[StorytellerCZ]

So, free hosting with Meteor React skeleton with the sdk only installed (nothing else added): manages to build and run
Adding in the settings and actually initializing one-sdk and asking for weather: fails

[lukas-valenta]

Hello @StorytellerCZ, we have now identified and removed most memory consuming part of the SDK's dependencies, Typescript. It is not yet released to the public as there are some potential problems we need to solve first, we'll let you know when it is. I have tried tiny Meteor container with a beta release and it worked properly without running out of memory. It was an interesting experience working with Meteor and I have also identified possible friction points in the future, such as file I/O and will keep it in mind in the future. Thank you for your report!

Here's some details if you're interested:
We used typescript-is for JSON validation, which brings runtime typescript dependency. This will be replaced with ajv and JSON-Schema validation.
The second point where we used typescript was our parser, which is needed by OneSDK, but not always. We decided to make it an optional peer dependency, which solves the typescript dependency, but as I'm sure you can imagine introduces it's own set of challenges.

[StorytellerCZ]

@lukas-valenta Awesome! Thanks a lot and keep me posted! I will test it once you release.

[StorytellerCZ]

Tried with the latest version (Meteor and Superface) and still getting some issues:

1qnz9
2022-07-15 19:07:33+02:001qnz9
2022-07-15 19:07:33+02:00<--- Last few GCs --->1qnz9
2022-07-15 19:07:33+02:001qnz9
2022-07-15 19:07:33+02:00[7:0x6442a90] 123600 ms: Mark-sweep 253.7 (257.7) -> 252.4 (257.2) MB, 3601.9 / 0.0 ms (average mu = 0.144, current mu = 0.052) allocation failure scavenge might not succeed1qnz9
2022-07-15 19:07:33+02:00[7:0x6442a90] 127403 ms: Mark-sweep 253.5 (258.0) -> 252.5 (257.2) MB, 3800.7 / 0.0 ms (average mu = 0.075, current mu = 0.001) allocation failure scavenge might not succeed1qnz9
2022-07-15 19:07:33+02:001qnz9
2022-07-15 19:07:33+02:001qnz9
2022-07-15 19:07:33+02:00<--- JS stacktrace --->1qnz9
2022-07-15 19:07:33+02:001qnz9
2022-07-15 19:07:33+02:00FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory1qnz9
2022-07-15 19:07:33+02:00 1: 0xa3aaf0 node::Abort() [node]1qnz9
2022-07-15 19:07:33+02:00 2: 0x970199 node::FatalError(char const*, char const*) [node]1qnz9
2022-07-15 19:07:33+02:00 3: 0xbba42e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]1qnz9
2022-07-15 19:07:33+02:00 4: 0xbba7a7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]1qnz9
2022-07-15 19:07:33+02:00 5: 0xd769c5 [node]1qnz9
2022-07-15 19:07:33+02:00 6: 0xd7754f [node]1qnz9
2022-07-15 19:07:33+02:00 7: 0xd8538b v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]1qnz9
2022-07-15 19:07:33+02:00 8: 0xd88f4c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]1qnz9
2022-07-15 19:07:33+02:00 9: 0xd5762b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]1qnz9
2022-07-15 19:07:33+02:0010: 0x109fbef v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]1qnz9
2022-07-15 19:07:33+02:0011: 0x1448df9 [node]

node v14.19.3
Meteor v2.7.3
@superfaceai/one-sdk v1.5.2

Removing superface allows me to deploy without problems.

[jnv]

@StorytellerCZ Thanks for an update! Unfortunately we still didn't get rid of dependency on TypeScript, since parser is still bundled. However, there's currently a refactoring initiative in progress, which will remove the parser from SDK. We plan to release it in v2.0 of SDK.

[StorytellerCZ]

Since v2 is out, I will try to take some time today to see if it fixes this issue.