Performance degraded for returning high number of rows #115
Comments
|
Does #113 address this issue? |
|
We are not using iex> {:ok, conn} = Ch.start_link
{:ok, #PID<0.240.0>}
iex> {us, :ok} = :timer.tc fn -> Ch.query(conn, "select * from system.numbers limit 40000"); :ok end
{8451, :ok} # 8.5ms
# the current stream implementation emits an element on each `Mint.HTTP1.recv(socket, 0, timeout)`
# here we have 4 elements, so we did 4 recvs
iex> Ch.run(conn, fn conn -> conn |> Ch.stream("select * from system.numbers limit 40000") |> Enum.into([]) end)
[
[
{:status, #Reference<0.3547367938.1947205634.96066>, 200},
{:headers, #Reference<0.3547367938.1947205634.96066>,
[
{"date", "Thu, 03 Aug 2023 15:25:59 GMT"},
{"connection", "Keep-Alive"},
{"content-type", "application/octet-stream"},
{"x-clickhouse-server-display-name", "mac3.local"},
{"transfer-encoding", "chunked"},
{"x-clickhouse-query-id", "93d3e063-84d5-4a4a-bf79-41af9a122034"},
{"x-clickhouse-format", "RowBinaryWithNamesAndTypes"},
{"x-clickhouse-timezone", "UTC"},
{"keep-alive", "timeout=100"},
{"x-clickhouse-summary",
"{\"read_rows\":\"40000\",\"read_bytes\":\"320000\",\"written_rows\":\"0\",\"written_bytes\":\"0\",\"total_rows_to_read\":\"0\",\"result_rows\":\"0\",\"result_bytes\":\"0\"}"}
]},
{:data, #Reference<0.3547367938.1947205634.96066>,
<<1, 6, 110, 117, 109, 98, 101, 114, 6, 85, 73, 110, 116, 54, 52>>}
],
[],
[
{:data, #Reference<0.3547367938.1947205634.96066>,
<<0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0,
3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, ...>>},
{:data, #Reference<0.3547367938.1947205634.96066>,
<<0, 64, 0, 0, 0, 0, 0, 0, 1, 64, 0, 0, 0, 0, 0, 0, 2, 64, 0, 0, 0, 0, 0,
0, 3, 64, 0, 0, 0, 0, 0, 0, 4, 64, 0, 0, 0, 0, 0, 0, 5, 64, ...>>},
{:data, #Reference<0.3547367938.1947205634.96066>,
<<0, 128, 0, 0, 0, 0, 0, 0, 1, 128, 0, 0, 0, 0, 0, 0, 2, 128, 0, 0, 0, 0,
0, 0, 3, 128, 0, 0, 0, 0, 0, 0, 4, 128, 0, 0, 0, 0, 0, 0, 5, ...>>}
],
[done: #Reference<0.3547367938.1947205634.96066>]
]
|
|
#113 relevance depends on what exactly your goals are. If you read raw data (like CSV) and don't need parsing, |
I don't think it would change much as everything that happens in that recursive call is cheap. You can check it with eprof for
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm facing an issue returing 40k lines.
When I reach clickhouse direct from http url it return's in matter of 1 second.
When I use CH as driver... it reachs a timeout from 35 seconds.
And from my debbug... I believe CH forces get line per line.
deps/ch/lib/ch/connection.ex:239in method namedrecvThe argument 0 bytes on recv forces just the next line.
Then CH go recursively getting more and more.
Suggestion:
maybe the number of bytes could be configured on request.The text was updated successfully, but these errors were encountered: