Source MySQL\MsSql\Postgres: added RDS base performance tests #8215

etsybaev · 2021-11-23T20:14:07Z

What

Need to test benchmarks

How

Added tests for MySql, MsSql and Postgres RDS DBs.

🚨 User Impact 🚨

No user impacts, tests only

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

etsybaev · 2021-11-29T13:24:33Z

/test-performance connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1516129500
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1516129500
🐛 https://gradle.com/s/bbyoelwbtnk2u
🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1516129500
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1516129500
🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1516129500
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1516129500

…of github.com:airbytehq/airbyte into akorotkov/source-perfomance-test-memory-and-cpu

andriikorotkov · 2021-12-10T13:47:46Z

@tuliren, I work on being able to pass CPU and memory values as gradle parameters for performance tests. for this I used this pull request from @etsybaev. I added my minor changes to this pull request, for buildSrc / src / main / groovy / airbyte-performance-test-java.gradle and airbyte-integrations / bases / standard-source-test / src / main / java / io / airbyte /integrations/standardtest/source/AbstractSourceConnectorTest.java

…rmance-test

tuliren

I see that there are still four databases we are testing against:

ID	Tables	Columns	Records	Purpose
D2	1K	240	200	Many tables.
D3	5K	240	200	Lots of tables.
D4	1K	8	10K	Many tables + moderate records.
D6	25	8	50K	Lots of records.

The cloud cannot handle 5000 tables currently. It fails at retrieving the catalog. Let's drop D3, and only run against D2 for now.

...rformance/java/io/airbyte/integrations/source/mssql/MsSqlRdsSourcePerformanceSecretTest.java

tuliren · 2021-12-11T00:00:15Z

Another thought indirectly related to this PR. Method AbstractSourceConnectorTest#runReadVerifyNumberOfReceivedMsgs mimics DefaultReplicationWorker#getReplicationRunnable to test the source connector.

It is vital that the former follows the same pattern as the latter. Is there a way to ensure that?

The latter method currently uses the sourceMessageTracker to track the stats of a sync on the source side. Ideally the former method should do the same.

Hi @tuliren . If I got you correctly - in the first case, we collect all received messages to collection and then verify that received messages are exactly as was expected. Meanwhile, such a direct approach wouldn't work for performance tests. We will quickly end up with outOfMemory (for the test itself thread, not for connector's) error if try to collect all messages to a collection. In the case of performance tests, we are not interested in how properly it was read, but we care about the stream name and number of messages. So the former approach checks more carefully but doesn't work as it is for cases with big DBs. Hope that was what you asked for. Please let me know if I got your question incorrectly. Thanks

I guess we are probably talking about different things. My first bullet point means the performance test should mimics the how the source is read as much as possible. Currently, we do that by replicating the production procedure (DefaultReplicationWorker#getReplicationRunnable) in AbstractSourceConnectorTest#runReadVerifyNumberOfReceivedMsgs. However, if the production procedure changes, the performance will not use the new procedure. So the test cannot truly reflect what happens in production. Ideally, the performance test should reference some of the production procedure directly, so that it is change-proof.

This can be worked on in a follow up PR though. Don't worry about it in this one.

I created an issue to track this: #8718.

.../airbyte/integrations/standardtest/source/performancetest/AbstractSourcePerformanceTest.java

etsybaev · 2021-12-12T13:19:26Z

/test-performance connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1569568144
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1569568144

etsybaev · 2021-12-12T13:19:36Z

/test-performance connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1569568506
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/1569568506

etsybaev · 2021-12-12T13:19:45Z

/test-performance connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1569568636
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/1569568636

etsybaev · 2021-12-12T15:15:12Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1569775596
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/1569775596
No Python unittests run

…ehq#8215) * Added RDS base performance tests for source-postgres, source-mssql and source-mysql * updated perfomance test with cpu and memory limit Co-authored-by: andriikorotkov <[email protected]>

Source-MySQL: added RDS base performance tests

cb4880c

github-actions bot added the area/connectors Connector related issues label Nov 23, 2021

etsybaev temporarily deployed to more-secrets November 23, 2021 20:14 Inactive

jrhizor temporarily deployed to more-secrets November 23, 2021 21:07 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 17:18 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 17:42 Inactive

jrhizor temporarily deployed to more-secrets November 24, 2021 17:48 Inactive

jrhizor temporarily deployed to more-secrets November 25, 2021 08:55 Inactive

jrhizor temporarily deployed to more-secrets November 25, 2021 15:58 Inactive

jrhizor temporarily deployed to more-secrets November 25, 2021 16:12 Inactive

removed $MYSQL_PERFORMANCE_TEST_CREDS config from github

61e80f6

etsybaev temporarily deployed to more-secrets November 25, 2021 16:31 Inactive

jrhizor temporarily deployed to more-secrets November 25, 2021 16:31 Inactive

jrhizor temporarily deployed to more-secrets November 25, 2021 16:39 Inactive

jrhizor temporarily deployed to more-secrets November 25, 2021 18:16 Inactive

etsybaev requested a review from alexandr-shegeda November 26, 2021 08:55

added few new tests

ef172ff

etsybaev temporarily deployed to more-secrets November 29, 2021 10:43 Inactive

jrhizor temporarily deployed to more-secrets November 29, 2021 11:17 Inactive

added few new test

2e19a54

etsybaev temporarily deployed to more-secrets November 29, 2021 12:07 Inactive

jrhizor temporarily deployed to more-secrets November 29, 2021 12:08 Inactive

fixed expected value

e8a3fcc

etsybaev temporarily deployed to more-secrets November 29, 2021 13:04 Inactive

jrhizor temporarily deployed to more-secrets November 29, 2021 13:26 Inactive

etsybaev temporarily deployed to more-secrets November 29, 2021 14:45 Inactive

jrhizor temporarily deployed to more-secrets November 29, 2021 16:01 Inactive

etsybaev requested review from tuliren and DoNotPanicUA November 29, 2021 16:04

andriikorotkov added 2 commits December 10, 2021 15:18

add logs

f256138

Merge branch 'etsybaev/source-mysql-added-base-rds-performance-test' …

747b7ed

…of github.com:airbytehq/airbyte into akorotkov/source-perfomance-test-memory-and-cpu

andriikorotkov temporarily deployed to more-secrets December 10, 2021 13:38 Inactive

andriikorotkov temporarily deployed to more-secrets December 10, 2021 13:40 Inactive

Merge branch 'master' into etsybaev/source-mysql-added-base-rds-perfo…

0c76263

…rmance-test

etsybaev temporarily deployed to more-secrets December 10, 2021 15:06 Inactive

etsybaev temporarily deployed to more-secrets December 10, 2021 22:34 Inactive

tuliren approved these changes Dec 10, 2021

View reviewed changes

tuliren mentioned this pull request Dec 11, 2021

Make sure the database benchmark test mimics the production #8718

Closed

updated tables rows count

414e009

andriikorotkov temporarily deployed to more-secrets December 11, 2021 15:49 Inactive

tuliren reviewed Dec 11, 2021

View reviewed changes

.../airbyte/integrations/standardtest/source/performancetest/AbstractSourcePerformanceTest.java Outdated Show resolved Hide resolved

updated code as per comments in PR

a360666

etsybaev temporarily deployed to more-secrets December 12, 2021 12:51 Inactive

jrhizor temporarily deployed to more-secrets December 12, 2021 13:21 Inactive

jrhizor temporarily deployed to more-secrets December 12, 2021 15:17 Inactive

etsybaev merged commit 70d2f46 into master Dec 12, 2021

etsybaev deleted the etsybaev/source-mysql-added-base-rds-performance-test branch December 12, 2021 17:29

tuliren mentioned this pull request Dec 12, 2021

[EPIC] Source DB Benchmark Tests #7035

Closed

6 tasks

jrhizor mentioned this pull request Dec 14, 2021

Bump Airbyte version from 0.33.11-alpha to 0.33.12-alpha #8787

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source MySQL\MsSql\Postgres: added RDS base performance tests #8215

Source MySQL\MsSql\Postgres: added RDS base performance tests #8215

etsybaev commented Nov 23, 2021 •

edited

Loading

etsybaev commented Nov 29, 2021 •

edited by github-actions bot

Loading

andriikorotkov commented Dec 10, 2021

tuliren left a comment •

edited

Loading

tuliren commented Dec 11, 2021 •

edited

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

Source MySQL\MsSql\Postgres: added RDS base performance tests #8215

Source MySQL\MsSql\Postgres: added RDS base performance tests #8215

Conversation

etsybaev commented Nov 23, 2021 • edited Loading

What

How

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

etsybaev commented Nov 29, 2021 • edited by github-actions bot Loading

andriikorotkov commented Dec 10, 2021

tuliren left a comment • edited Loading

Choose a reason for hiding this comment

tuliren commented Dec 11, 2021 • edited Loading

etsybaev commented Dec 12, 2021 • edited by github-actions bot Loading

etsybaev commented Dec 12, 2021 • edited by github-actions bot Loading

etsybaev commented Dec 12, 2021 • edited by github-actions bot Loading

etsybaev commented Dec 12, 2021 • edited by github-actions bot Loading

etsybaev commented Nov 23, 2021 •

edited

Loading

etsybaev commented Nov 29, 2021 •

edited by github-actions bot

Loading

tuliren left a comment •

edited

Loading

tuliren commented Dec 11, 2021 •

edited

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading

etsybaev commented Dec 12, 2021 •

edited by github-actions bot

Loading