Investigate memory usage when using chaos_all_columns table with 100k rows

[pskrbasu]

When using the chaos_all_columns tables, collecting about 100,000 rows, there is a significant surge in memory usage by tailpipe.

partition config:

partition "chaos_all_columns" "high_row_count" {
  source "chaos_all_columns" {
    row_count = 100000
  }
}

Observations:

• When the collect process starts, tailpipe memory usage surges up to 8GB.
• Then drops to almost 2GB and then steadily increases(up to 3.5GB) during the rows converted.
• This memory usage for chaos is extremely high compared to other plugins(AWS).

Rows	Max	Start
10k	1.12G	700-800M
20k	1.91G	1.3-1.4G
50k	3.16G	2.06G
100k	3.32G (10G)	1.92G

Investigate why the surge happens and also why chaos uses more mem than other plugins. Is it because of how the plugin is designed?