Buffer Overflow?

[wilwade]

Acknowledgements

• I read the documentation and searched existing issues to avoid duplicates
• I understand this is a bug tracker and anything other than a proven bug will be closed
• I understand this is a free project and relies on community contributions
• I read and understood the Contribution guide

Minimal reproduction URL

https://gist.github.com/wilwade/66bc4614fdaa3a349c5f25b7164d12bc

Problem & expected behavior (under 200 words)

tsx Version: v4.19.1, v4.10.2
Note: This ALSO happens with ts-node v10.9.2

tldr: When using a buffer, the buffer.buffer is getting filled up with stuff it should not be.

let decoder = new TextDecoder("utf-8");
const zerosArray = new Array(1024).fill(0);
// @ts-ignore
const buf = Buffer.from(zerosArray);

Bugs are expected to be fixed by those affected by it

• I'm interested in working on this issue

Compensating engineering work will speed up resolution and support the project

• I'm willing to offer $10 for financial support

[shannonwells]

More info:

• the behavior also occurs for Buffer.copyBytesFrom
• the Buffer's ArrayBuffer is always 8192 in length, with a byteLength & length of whatever the original array length was, and a byteOffset = 4568 or something like it.
• Instance method toString() outputs the correct values anyway
• Instance method entries() also outputs the correct values
• Does not occur when the same code is compiled into JS and then executed.

The workaround is to use Buffer.alloc with a TypedArray. Example:

const uint8array = new Uint8Array([0,1,2,3,5,8,13]);
const buf = Buffer.alloc(uint8array.length, uint8array)

The above results in the expected values in the Buffer's ArrayBuffer, with a length and byteLength of 7, byteOffset = 0 as expected.

[fallaciousreasoning]

Okay, this seems to be related to Buffer.poolSize - the only reason this comes up in ts-node and tsx and not in the following JS:

const buf1 = Buffer.from('hello')
console.log(new TextDecoder().decode(buf1.buffer))
// outputs "hello"

is because there's no other data in a buffer in that JS - its pretty straightforward to duplicate with the following:

const buf0 = Buffer.from("I'm some other text that you didn't expect!")
const buf1 = Buffer.from('hello')
console.log(new TextDecoder().decode(buf1.buffer))
// outputs "/I'm some other text that you didn't expect!hello"

You can prevent this from happening with (what I'm sure is a pretty significant performance penalty) but setting Buffer.poolSize = 1 before creating your buffer:

const buf0 = Buffer.from("I'm some other text that you didn't expect!")
Buffer.poolSize = 1
const buf1 = Buffer.from('hello')
console.log(new TextDecoder().decode(buf1.buffer))
// outputs "hello"

I think this might just be working as intended because buffer.buffer is shared across multiple Buffers (which is why there's an byteOffset and length on Buffer), presumably for performance.

I think this issue can probably be closed (even though the behavior is pretty counterintuitive).