Data spill to the next chunk unexpectedly

[michaelfung]

I use chunked transfer with code like the documentation. When transferring thousands of lines of JSON string e.g. {"user_id":123456}. I send 100 lines per chunk and each line is less than 20 bytes.

The problem is: at the receiving end (use Mojo::UserAgent) randomly some lines failed to decode as they are split between chunks. So, is this behavior normal or a bug?

# Sending side:
my $drain = sub {
  my $c = shift;
  my $chunk = '';
  for ( 1 .. 100 ) {
      last unless my $row = $users[$i++];
      $chunk .= encode_json( $row ) . "\n";
  }
  return $c->finish if ($chunk eq '');
  $c->write_chunk( $chunk, __SUB__ );
};
$c->$drain;

# Receving side:
$tx->res->content->unsubscribe('read')->on(read => sub ($mojo_content_obj, $chunk) {
    my $chunk_row = 0;
    open my $fh, '<', \$chunk;
    while (my $line = <$fh>) {
      ++$chunk_row;
      my $row = eval { decode_json $line; };
      warn "bad row at beginning of chunk, line=$line\n" if $@ and $chunk_row == 1;
      ...
  }

[Grinnz]

If you do a chunked read you have to buffer the received bytes until you receive a full JSON object to decode

[michaelfung]

According to the specification, there will be size in the head and a delimiter \r\n at the end of each chunk, I think we can make use of it to tell a complete chunk.

[Grinnz]

Ah, you are referring to chunked transfer encoding. Perhaps you are being mislead by the use of the word "chunk" in the read event description, but it operates on each chunk of bytes that arrives, unrelated to transfer encoding.

[michaelfung]

I would like to know what is the rationale for not doing emit "read" event only when whole chunk(s) is received?

It works for my use case after the following patch to Mojo::Content.

--- Content.pm.orig     2022-01-13 06:06:09.354216441 +0000
+++ Content.pm  2022-01-14 02:11:24.701287118 +0000
@@ -102,9 +102,11 @@
 
   # Chunked or relaxed content
   if ($self->is_chunked || $self->relaxed) {
-    $self->_decompress($self->{buffer} //= '');
+    if ($self->{whole_chunk_read}) {
+      $self->_decompress($self->{buffer} //= '') ;
+      $self->{buffer} = '';
+    }
     $self->{size} += length $self->{buffer};
-    $self->{buffer} = '';
     return $self;
   }
 
@@ -219,6 +221,7 @@
     $self->{buffer} .= substr $self->{pre_buffer}, 0, $len, '';
     $self->{real_size} += $len;
     $self->{chunk_len} -= $len;
+    $self->{whole_chunk_read} = ($self->{chunk_len} == 0) ? 1 : 0;
   }

[Grinnz]

Most HTTP content is not sent using chunked transfer encoding.

[michaelfung]

Hmm...??? Thanks anyway!

[michaelfung]

For the record:

it operates on each chunk of bytes that arrives, unrelated to transfer encoding.