Replies: 1 comment
-
The Postgres connector is single threaded for reads. How much throughput does a SELECT * over JDBC give you on the source database? I'd assume it's 50MBps that you area observing. In other words how much time does it take to do a |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
I am exporting data from PostgreSQL to another database using the following query:
INSERT INTO db_name.public.example_table (id, name, content, boolean, timestamp) SELECT id, name, content, CAST(boolean AS tinyint), CAST(timestamp AS timestamp(0)) FROM postgresql.public.example_table;
However, the transfer speed is always limited to around 50 MB per second. Here are the details of my setup:
Ping between hosts: 0.6 ms
Network bandwidth: 1 Gbps
PostgreSQL server specs: Intel Cascade Lake, 8 CPUs, 8 GB RAM
Target database server specs: Similar to PostgreSQL server.
Trino server: Intel Cascade Lake 16 CPU 32 GB RAM.
I have set the following configurations in Trino:
SET SESSION task_writer_count = 128; SET SESSION task_concurrency = 128; SET SESSION query_max_memory = '10GB'; SET SESSION query_max_total_memory = '10GB'; SET SESSION query_max_memory_per_node = '4GB';
Despite these settings, the transfer speed does not improve. I am 100% sure the issue is not with the target database, as I have tried multiple different setups. Additionally, during the export process, resource utilization never exceeds 50%. What could be causing this bottleneck and how can I optimize the data transfer speed?"
Beta Was this translation helpful? Give feedback.
All reactions