RAG Knowledge JavaScript Heap Out of Memory #3664

suryanshkushwaha · 2025-02-25T09:14:41Z

Description

The application is crashing with a "JavaScript heap out of memory" error when processing knowledge/messages. The error occurs during runtime execution with a heap size of approximately 4GB.

To Reproduce

Run the application with Node.js v23.8.0/v23.3.0
Process single path or single directory knowledge
Application crashes with heap out of memory error

Error Details

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Stack Trace

Key components in stack trace:

Heap allocation failure during garbage collection
Error occurs during array push operations
Triggered during async function processing and microtask execution

Technical Details

Node.js Version: v23.8.0/v23.3.0
Platform: macOS
Current Memory Usage: ~4075MB before crash

Proposed Solutions

Unaware

Additional Context

Error occurs during knowledge processing with ragKnowledge enabled ONLY

Error

[2025-02-25 08:58:52] INFO: [Timing] Main embedding: 0.15s

<--- Last few GCs --->

[11338:0x140008000]    18160 ms: Scavenge (interleaved) 4075.1 (4085.0) -> 4075.1 (4108.0) MB, pooled: 0 MB, 20.17 / 0.00 ms  (average mu = 0.179, current mu = 0.138) allocation failure; 
[11338:0x140008000]    19484 ms: Mark-Compact (reduce) 4075.8 (4108.0) -> 4075.8 (4078.8) MB, pooled: 0 MB, 1188.88 / 0.00 ms  (+ 9.5 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1202 ms) (average mu = 0.160, 

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: node::OOMErrorHandler(char const*, v8::OOMDetails const&) 
 2: v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) 
 3: v8::internal::Heap::stack() 
 4: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_1::operator()() const 
 5: void heap::base::Stack::SetMarkerAndCallbackImpl<v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags)::$_1>(heap::base::Stack*, void*, void const*) 
 6: PushAllRegistersAndIterateStack 
 7: v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) 
 8: v8::internal::StackGuard::HandleInterrupts(v8::internal::StackGuard::InterruptLevel) 
 9: v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) 
10: Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit 
11: Builtins_ArrayPrototypePush 
12:  
13: Builtins_InterpreterEntryTrampoline 
14: Builtins_InterpreterEntryTrampoline 
15: Builtins_AsyncFunctionAwaitResolveClosure 
16: Builtins_PromiseFulfillReactionJob 
17: Builtins_RunMicrotasks 
18: Builtins_JSRunMicrotasksEntry 
19: v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) 
20: v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) 
21: v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) 
22: v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) 
23: v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) 
24: node::InternalCallbackScope::Close() 
25: node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) 
26: node::InternalMakeCallback(v8::Isolate*, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context, v8::Local<v8::Value>) 
27: node::Environment::CheckImmediate(uv_check_s*) 
28: uv__run_check 
29: uv_run 
30: node::SpinEventLoopInternal(node::Environment*) 
31: node::NodeMainInstance::Run() 
32: node::Start(int, char**) 
33: start 
sh: line 1: 11338 Abort trap: 6           node --loader ts-node/esm src/index.ts --characters=./characters/kaiadevbot.character.json
 ELIFECYCLE  Command failed with exit code 134.

The text was updated successfully, but these errors were encountered:

github-actions · 2025-02-25T09:15:08Z

Hello @suryanshkushwaha! Welcome to the elizaOS community. Thank you for opening your first issue; we appreciate your contribution. You are now an elizaOS contributor!

metatxn · 2025-02-25T10:14:00Z

@suryanshkushwaha try this. it worked for me.

export NODE_OPTIONS="--max-old-space-size=4096"

boolkeys · 2025-02-25T10:38:05Z

@suryanshkushwaha try this. it worked for me.
export NODE_OPTIONS="--max-old-space-size=4096"

that also worked for me.

If you want to set a different memory size, you can adjust the number:
2GB: --max-old-space-size=2048
6GB: --max-old-space-size=6144
8GB: --max-old-space-size=81924

myconite · 2025-02-25T12:13:07Z

Same issue.

@boolkeys or @metatxn, where to set the export NODE_OPTIONS="--max-old-space-size=4096"?

boolkeys · 2025-02-25T12:15:26Z

Same issue.

@boolkeys or @metatxn, where to set the export NODE_OPTIONS="--max-old-space-size=4096"?

Just run it in CLI in the same place where you run pnpm commands

darab-ui · 2025-02-25T13:04:08Z

افزایش حافظه: دستور: `node --max-old-space-size=8192 --loader ts-node/esm src/index.ts --characters=./characters/kaiadevbot.character.json 8192 به معنای ۸ گیگابایت حافظه است. مدیریت حافظه: استفاده کمتر از آرایه‌ها و اشیاء بزرگ، استفاده از جریان‌های داده (Streams)بهینه‌سازی داده‌های غیرهمزمان، تقسیم داده‌ها به بخش‌های کوچک‌تر

myconite · 2025-02-25T13:06:14Z

@boolkeys did it in CLI with command:

pnpm start --characters="characters/default.character.json" --max-old-space-size=8192

but it's not working for me :/

<--- Last few GCs --->

[35053:0x2f83c000] 27258 ms: Scavenge 1949.1 (1966.9) -> 1949.1 (1981.9) MB, pooled: 0 MB, 40.30 / 0.00 ms (average mu = 0.385, current mu = 0.245) allocation failure;
[35053:0x2f83c000] 34786 ms: Mark-Compact 2537.9 (2555.7) -> 2155.1 (2188.4) MB, pooled: 0 MB, 3447.22 / 0.00 ms (average mu = 0.527, current mu = 0.594) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

----- Native stack trace -----

1: 0xe7e015 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0x12e8be0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0x12e8f7c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x154be55 [node]
5: 0x155f262 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
6: 0x1537d82 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x15380b8 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x150f340 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
9: 0x199136b v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x71fe4666f276
Aborted (core dumped)
/home/urban/projects/eliza/agent:
ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL @elizaos/[email protected] start: node --loader ts-node/esm src/index.ts "--isRoot" "--characters=characters/default.character.json" "--max-old-space-size=8192"
Exit status 134
ELIFECYCLE Command failed with exit code 134.

boolkeys · 2025-02-25T13:44:16Z

@boolkeys did it in CLI with command:

pnpm start --characters="characters/default.character.json" --max-old-space-size=8192

but it's not working for me :/

<--- Last few GCs --->

[35053:0x2f83c000] 27258 ms: Scavenge 1949.1 (1966.9) -> 1949.1 (1981.9) MB, pooled: 0 MB, 40.30 / 0.00 ms (average mu = 0.385, current mu = 0.245) allocation failure;
[35053:0x2f83c000] 34786 ms: Mark-Compact 2537.9 (2555.7) -> 2155.1 (2188.4) MB, pooled: 0 MB, 3447.22 / 0.00 ms (average mu = 0.527, current mu = 0.594) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

----- Native stack trace -----

1: 0xe7e015 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
2: 0x12e8be0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
3: 0x12e8f7c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
4: 0x154be55 [node]
5: 0x155f262 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
6: 0x1537d82 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
7: 0x15380b8 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
8: 0x150f340 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
9: 0x199136b v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x71fe4666f276
Aborted (core dumped)
/home/urban/projects/eliza/agent:
ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL @elizaos/[email protected] start: node --loader ts-node/esm src/index.ts "--isRoot" "--characters=characters/default.character.json" "--max-old-space-size=8192"
Exit status 134
ELIFECYCLE Command failed with exit code 134.

No, just type it as a new command, not with pnpm start. Just type in CLI:

export NODE_OPTIONS="--max-old-space-size=4096"

myconite · 2025-02-25T13:56:47Z

@boolkeys cheers, javascript is not something I use often. That seems to have worked and was applied. However my host machine (vm) then crashed later after trying to run the character and apply rag knowledge. I have 6 gb allocated to it in total. Seems like there was too much memory consumption...

darab-ui · 2025-02-25T14:06:57Z

برای بهینه‌سازی مصرف حافظه در Node.js، ابتدا باید بخش‌های پرمصرف را شناسایی کنیم. سپس از روش‌هایی مانند استفاده از جریان‌ها (Streams)، پردازش داده‌ها به بخش‌های کوچکتر، و استفاده از Garbage Collection دستی برای کاهش مصرف حافظه بهره ببریم

DEVMYTH123 · 2025-02-25T15:07:03Z

same issue even using --max-old-space-size=8192 and even 16384 which is 16GB of ram. Idts it is the issue of the less ram.

darab-ui · 2025-02-25T17:24:51Z

تحلیل و بهینه سازی node --inspect --loader ts-node/esm src/index.ts --characters=./characters/default.character.json،از Garbage collectionبرای جمع اوری زباله و اینکه فواصل بین عملیات را افزایش دهد تا فرصت بیشتری برای پردازش داده ها باشد ،بررسی الگوریتم های بهینه تز تا حافظه کمتری استفاده شود ،پروفایلتان و مانیتورینگ و اینکه کدام بخش ها نیار به بهینه سازی دارد clinic.jsیاheapdumpو اینکه تقسیم فرایندها به بخش های کوچک تر

یا اینکه کلا
const fs = require('fs');
const readline = require('readline');
const stream = require('stream');
const { program } = require('commander');

// جریان تبدیل برای پردازش خطی
class LineTransform extends stream.Transform {
constructor() {
super({ readableObjectMode: true });
}

_transform(chunk, encoding, callback) {
const lines = chunk.toString().split('\n');
lines.forEach(line => {
const cleanedLine = line.trim();
if (cleanedLine.length > 0) {
this.push(cleanedLine);
}
});
callback();
}
}

async function processFile(filePath) {
try {
if (!fs.existsSync(filePath)) {
console.error(❌ خطا: فایل "${filePath}" یافت نشد.);
return;
}

const fileStream = fs.createReadStream(filePath, { highWaterMark: 64 * 1024 }); // بافر 64KB
fileStream.on('error', (err) => console.error(`❌ خطا در باز کردن فایل: ${err.message}`));

const lineTransform = new LineTransform();
let lineCount = 0;

lineTransform.on('data', (line) => {
  console.log(`پردازش خط: ${line}`);
  lineCount++;
  if (lineCount % 50 === 0) {
    console.log(`📊 ${lineCount} خط پردازش شد`);
  }
});

lineTransform.on('end', () => {
  console.log('✅ پردازش فایل با موفقیت انجام شد');
});

fileStream.pipe(lineTransform);

} catch (error) {
console.error('❌ خطا در پردازش فایل:', error);
}
}

// تنظیمات CLI
program
.version('1.0.0')
.description('برنامه پردازش فایل')
.argument('', 'فایل ورودی')
.action((file) => processFile(file));

program.parse(process.argv);

t-phoenix · 2025-02-25T21:58:05Z

Facing the same issue, using MAC M2
tried export NODE_OPTIONS="--max-old-space-size=4096" don't seem to be working for me.
Edited: Worked export NODE_OPTIONS="--max-old-space-size=6144" for me

myconite · 2025-02-26T02:25:08Z

Some more info from testing:

I loaded up 200+ .md files (each one is an article) and getting a lot of:

Failed to read knowledge file {removed_actual_file_name}.md. Error details: Invalid array length

I'm using --max-old-space-size=6144 and I tried to pnpm start my character about 20 times now. It brakes down every time, but it does process some new files in the meantime.

suryanshkushwaha · 2025-02-26T04:35:37Z

@myconite Same issue for me, the export option seemed to work but now I am also getting the same error

t-phoenix · 2025-02-26T16:51:38Z

@myconite and @suryanshkushwaha, please do double check unnecessary line breaks and spaces. I was getting the same error due to slightly mismanaged prompt format.

myconite · 2025-02-27T03:17:28Z

@t-phoenix no it's not that CLI command is fine. It seems to have something to do with the amount of files that need processing or the memory management of it? I ran the CLI command with the character and rag knowledge now about 40 times probably. Each time it does add some new articles to the database. Eventually it will probably go through them and finally load the character... until then I am still getting a lot of these:

Failed to read knowledge file {removed_actual_file_name}.md. Error details: Invalid array length

As can be seen here in the logs:

[2025-02-27 03:21:24] INFO: Knowledge {removed_actual_file_name}.md unchanged, skipping
[2025-02-27 03:21:24] INFO: Processing MD file content for {removed_actual_file_name}.md
[2025-02-27 03:21:24] INFO: [File Progress] Starting {removed_actual_file_name}.md (21.08 KB)
[2025-02-27 03:21:24] INFO: [Timing] Preprocessing: 0.00s
[2025-02-27 03:21:24] INFO: Knowledge {removed_actual_file_name}.md unchanged, skipping
[2025-02-27 03:21:24] INFO: Processing MD file content for {removed_actual_file_name}.md
[2025-02-27 03:21:24] INFO: [File Progress] Starting {removed_actual_file_name}.md (5.52 KB)
[2025-02-27 03:21:24] INFO: [Timing] Preprocessing: 0.00s
[2025-02-27 03:21:24] INFO: [Timing] Main embedding: 0.47s
[2025-02-27 03:21:24] INFO: [Timing] Main document storage: 0.49s
[2025-02-27 03:21:50] ERROR: Error processing file {removed_actual_file_name}.md:
[2025-02-27 03:21:50] ERROR: Failed to read knowledge file {removed_actual_file_name}.md. Error details: Invalid array length
[2025-02-27 03:21:50] WARN: Some knowledge items failed to process, but continuing with available knowledge
[2025-02-27 03:21:50] INFO: [Timing] Main embedding: 25.97s

suryanshkushwaha added the bug Something isn't working label Feb 25, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RAG Knowledge JavaScript Heap Out of Memory #3664

RAG Knowledge JavaScript Heap Out of Memory #3664

suryanshkushwaha commented Feb 25, 2025

github-actions bot commented Feb 25, 2025

metatxn commented Feb 25, 2025

boolkeys commented Feb 25, 2025

myconite commented Feb 25, 2025

boolkeys commented Feb 25, 2025

darab-ui commented Feb 25, 2025

myconite commented Feb 25, 2025 •

edited

Loading

boolkeys commented Feb 25, 2025

myconite commented Feb 25, 2025

darab-ui commented Feb 25, 2025

DEVMYTH123 commented Feb 25, 2025

darab-ui commented Feb 25, 2025

t-phoenix commented Feb 25, 2025 •

edited

Loading

myconite commented Feb 26, 2025

suryanshkushwaha commented Feb 26, 2025

t-phoenix commented Feb 26, 2025

myconite commented Feb 27, 2025 •

edited

Loading

RAG Knowledge JavaScript Heap Out of Memory #3664

RAG Knowledge JavaScript Heap Out of Memory #3664

Comments

suryanshkushwaha commented Feb 25, 2025

Description

To Reproduce

Error Details

Stack Trace

Technical Details

Proposed Solutions

Additional Context

Error

github-actions bot commented Feb 25, 2025

metatxn commented Feb 25, 2025

boolkeys commented Feb 25, 2025

myconite commented Feb 25, 2025

boolkeys commented Feb 25, 2025

darab-ui commented Feb 25, 2025

myconite commented Feb 25, 2025 • edited Loading

boolkeys commented Feb 25, 2025

myconite commented Feb 25, 2025

darab-ui commented Feb 25, 2025

DEVMYTH123 commented Feb 25, 2025

darab-ui commented Feb 25, 2025

t-phoenix commented Feb 25, 2025 • edited Loading

myconite commented Feb 26, 2025

suryanshkushwaha commented Feb 26, 2025

t-phoenix commented Feb 26, 2025

myconite commented Feb 27, 2025 • edited Loading

myconite commented Feb 25, 2025 •

edited

Loading

t-phoenix commented Feb 25, 2025 •

edited

Loading

myconite commented Feb 27, 2025 •

edited

Loading