Streaming writes of large files on response

[joshxyzhimself]

Let's say I have a large file from 1gb to 10gb.

Couple questions:

1. I don't think I can pipe fs.createReadStream to res? My understanding is I cant. I've found Posible to stream gzip content? #462 but I don't think pipe will work there?
2. I think I could fs.createReadSteam(path).on('data', (chunk) => res.write(chunk)); but what if the client has slow download speed, will the res.write(chunk) build-up on the server-side?
3. I think I could fd = fs.open(path) to get a file descriptor, then create a handler at res.onWritable((offset) => res.write(fs.read(fd, { offset }))), this way when onWritable handler is called, I read some bytes then write those bytes?

Is my understanding correct that option 3 will work?

Thank you.

[ghost]

There is an example for this: https://github.com/uNetworking/uWebSockets.js/blob/master/examples/VideoStreamer.js

Only something like 3 will work. 1 and 2 are No/Don't.

[joshxyzhimself]

Noted on these, thank you.

I've noticed that res.tryEnd(recognized_string, total_size) requires the second parameter, this means I can only use res.tryEnd() and res.onWritable() with ReadableStreams where I know the value for the total_size.

Examples:

1. ReadableStream of files in file system, where I can use fs.stat to get the file size.
2. ReadableStream of incoming HTTP requests (or responses from outgoing HTTP requests) where there is Content-Length header.

I think the total_size here is being used for the response Content-Length header, and for deciding if uws should call another onWritable handler or if it should end it?

However on the following use cases which are on-the-fly modification of data, I don't think I can have the value for total_size beforehand:

1. ReadableStreams that pass through compression, such as fs.createReadStream(path).pipe(zlib.createGzip()) or fs.createReadStream(path).pipe(zlib.createBrotliCompress()) streams. (https://nodejs.org/api/zlib.html)
2. ReadableStreams that pass through image processing, such as fs.createReadStream(path).pipe(sharp().jpeg({ mozjpeg: true })). (https://github.com/lovell/sharp)

I'm thinking of something similar to the following flow:

const readable_stream = fs.createReadStream(file_path).pipe(zlib.createGzip());
readable_stream.on('data', (chunk_buffer) => {
  // chunk_buffer here is 64 * 1024 bytes by default since we are using fs.createReadStream
  res.write(chunk_buffer);
  readable_stream.pause();
});
readable_stream.on('end', () => {
  res.end();
});
res.onWritable(() => {
  // instead of using offset, just resume the stream, write the next chunk, then pause the stream
  readable_stream.resume();
});

[ghost]

That's almost correct but you cannot pause readable_stream unless res.write returns a failure. Otherwise you'll end up in limbo where no progress is made.

[joshxyzhimself]

Hmmm but res.write(recognized_string) only returns a boolean on whether no backpressure was added or not; not if it is a failure. Will it throw an error on that line if it fails?

On the code below I pause if there is backpressure, then just resume on the onWritable handler.

I've ran the following code on a 600mb (5-million-lines) csv file and miraculously the result was okay (same amount of bytes / readable / doesn't seem corrupt). Also tested it on a 420kb png file and result was okay. Do you see any problems with it?

readable_stream.on('data', (buffer) => {

  // will this line throw if res.write(buffer) fails? can wrap it in try-catch
  const no_backpressure = res.write(buffer);

  if (no_backpressure === false) {
    readable_stream.pause();
  }

});

readable_stream.on('end', () => {
  res.end();
});

res.onWritable(() => {
  readable_stream.resume();
  return true;
});