Sync seems to get stuck for a while

[sw-lkinsella]

We are currently running Airbyte on AKS and are syncing Postgres to Snowflake. We have split up the sync into multiple sources so as to run some things in parallel (there's no cross over and they have their own replication slots).

Most of the syncs run fine, however for one of them we are seeing this a lot:

2024-07-16 08:52:31 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:52:41 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:52:51 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:53:01 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:53:11 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:53:21 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:53:25 destination > INFO pool-34-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2024-07-16 08:53:25 destination > INFO pool-37-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2024-07-16 08:53:31 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:53:42 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:53:52 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:54:02 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:54:12 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:54:22 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:54:25 destination > INFO pool-34-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2024-07-16 08:54:25 destination > INFO pool-37-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0 2024-07-16 08:54:32 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:54:42 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:54:52 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:55:02 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:55:13 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:55:23 source > INFO pool-2-thread-1 i.a.c.i.d.AirbyteDebeziumHandler$CapacityReportingBlockingQueue(reportQueueUtilization):49 CDC events queue size: 0. remaining 10000 2024-07-16 08:55:25 destination > INFO pool-34-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):94 [ASYNC QUEUE INFO] Global: max: 742.41 MB, allocated: 10 MB (10.0 MB), %% used: 0.013469714189502041 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0 2024-07-16 08:55:25 destination > INFO pool-37-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0

It goes on like this for lines and lines. Eventually the sync does complete (takes about an hour) so I'm wondering what is causing this and how to fix it?

I'm hoping someones seen this before or has an idea.

[cbuckle1]

Airbyte on ec2, with a sync from Salesforce to Snowflake and we are seeing the same thing - any resolution?
State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.00...

[taherajnaFA]

Airbyte on ec2 with sync from Postgres source to Postgres destination. The below eats up the majority of the sync time without doing anything meaningful:

2024-08-23 04:45:31 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-08-23 04:46:31 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-08-23 04:46:31 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-08-23 04:47:31 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-08-23 04:47:31 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0
2024-08-23 04:48:31 destination > INFO pool-3-thread-1 i.a.c.i.d.a.b.BufferManager(printQueueInfo):89 [ASYNC QUEUE INFO] Global: max: 296.96 MB, allocated: 10 MB (10.0 MB), %% used: 0.03367428551701215 | State Manager memory usage: Allocated: 10 MB, Used: 0 bytes, percentage Used 0.0
2024-08-23 04:48:31 destination > INFO pool-6-thread-1 i.a.c.i.d.a.FlushWorkers(printWorkerInfo):127 [ASYNC WORKER INFO] Pool queue size: 0, Active threads: 0