Run tests for different connectors in different builds #10440

arhimondr · 2018-04-18T18:04:29Z

We were observing a lot of memory related intermittent product test failures on master. In order to test statistics API we are required to run ANALYZE on hive, that takes us some additional memory to run.

The idea is to move mysql, cassandra, posgree tests to a separate build. So there is no need to provision both, memory demanding cassandra and hive for the same build.

This allows us to avoid spinning up all backends (hadoop, mysql, cassandra, posgree)
alltogether to save some memory

This is just a temporary solution to simply fix intermittent failutes.

Eventually we would like to have something like this: #10439

electrum

Looks good to me, but I'll leave it to someone more familiar with product tests to accept

kokosing

Can you also make Presto to use INFO for logging?

kokosing · 2018-04-18T18:40:18Z

.travis.yml

@@ -73,7 +73,7 @@ script:
  - |
    if [[ -v PRODUCT_TESTS_BASIC_ENVIRONMENT ]]; then
      presto-product-tests/bin/run_on_docker.sh \
-        multinode -x quarantine,big_query,storage_formats,profile_specific_tests,tpcds
+        multinode -x quarantine,big_query,storage_formats,profile_specific_tests,tpcds,cassandra,mysql_connector,postgresql_connector,mysql


why do we have mysql and mysql_connector? It would be nice to merge them. Maybe something for following PR.

Yeah, i noticed that. We should definitely do that as part of the follow up.

please file a ticket for that, to not to forget about it

#10439 (comment)

kokosing · 2018-04-18T18:43:50Z

presto-product-tests/bin/run_on_docker.sh

  EXTERNAL_SERVICES="hadoop-master sqlserver"
 elif [[ "$ENVIRONMENT" == "singlenode-ldap" ]]; then
  EXTERNAL_SERVICES="hadoop-master ldapserver"
+elif [[ "$ENVIRONMENT" == "singlenode-mysql" ]]; then
+  EXTERNAL_SERVICES="hadoop-master mysql"


do we need hadoop when testing mysql?

We don't. But due to the limitation in Tempto related to convention based tests, no matter what test suited you select it will alwasy try to provision TPCH dataset in Hive. We should fix that as part of the followup.

Can you file a ticket for tempto for that? I will try to fix that.

prestodb/tempto#263

kokosing · 2018-04-18T18:44:05Z

presto-product-tests/bin/run_on_docker.sh

+elif [[ "$ENVIRONMENT" == "singlenode-mysql" ]]; then
+  EXTERNAL_SERVICES="hadoop-master mysql"
+elif [[ "$ENVIRONMENT" == "singlenode-postgresql" ]]; then
+  EXTERNAL_SERVICES="hadoop-master postgresql"


do we really need hadoop when testing psql?

kokosing · 2018-04-18T18:44:41Z

presto-product-tests/bin/run_on_docker.sh

+elif [[ "$ENVIRONMENT" == "singlenode-postgresql" ]]; then
+  EXTERNAL_SERVICES="hadoop-master postgresql"
+elif [[ "$ENVIRONMENT" == "singlenode-cassandra" ]]; then
+  EXTERNAL_SERVICES="hadoop-master cassandra"


and here do we need hadoop when testing cassandra? I would prefer to use tpch if any connector has to be used other than cassandra.

same problem

kokosing · 2018-04-18T18:46:44Z

presto-product-tests/conf/docker/singlenode-kerberos-hdfs-no-impersonation/compose.sh

@@ -7,6 +7,5 @@ source "${BASH_SOURCE%/*}/../common/compose-commons.sh"
 docker-compose \
    -f ${BASH_SOURCE%/*}/../common/standard.yml \
    -f ${BASH_SOURCE%/*}/../common/kerberos.yml \
-    -f ${BASH_SOURCE%/*}/../common/jdbc_db.yml \


you haven't changed tests suites for this this environment, are you sure that it was not using jdbc connectors?

We used to start jdbc services only for multinode and singlenode environments. This include was simply extra / copy-paste artifact.

https://github.com/prestodb/presto/blob/master/presto-product-tests/bin/run_on_docker.sh#L145

kokosing · 2018-04-18T18:46:47Z

presto-product-tests/conf/docker/singlenode-ldap/compose.sh

@@ -6,6 +6,5 @@ source "${BASH_SOURCE%/*}/../common/compose-commons.sh"

 docker-compose \
    -f ${BASH_SOURCE%/*}/../common/standard.yml \
-    -f ${BASH_SOURCE%/*}/../common/jdbc_db.yml \


you haven't changed tests suites for this this environment, are you sure that it was not using jdbc connectors?

Although jdbc_db was included it was never provisioned

https://github.com/prestodb/presto/blob/master/presto-product-tests/bin/run_on_docker.sh#L150

kokosing · 2018-04-18T18:46:55Z

presto-product-tests/conf/docker/singlenode-kerberos-hdfs-impersonation/compose.sh

@@ -7,6 +7,5 @@ source "${BASH_SOURCE%/*}/../common/compose-commons.sh"
 docker-compose \
    -f ${BASH_SOURCE%/*}/../common/standard.yml \
    -f ${BASH_SOURCE%/*}/../common/kerberos.yml \
-    -f ${BASH_SOURCE%/*}/../common/jdbc_db.yml \


you haven't changed tests suites for this this environment, are you sure that it was not using jdbc connectors?

kokosing · 2018-04-18T18:47:02Z

presto-product-tests/conf/docker/singlenode-hdfs-impersonation/compose.sh

@@ -6,6 +6,5 @@ source "${BASH_SOURCE%/*}/../common/compose-commons.sh"

 docker-compose \
    -f ${BASH_SOURCE%/*}/../common/standard.yml \
-    -f ${BASH_SOURCE%/*}/../common/jdbc_db.yml \


you haven't changed tests suites for this this environment, are you sure that it was not using jdbc connectors?

kokosing · 2018-04-18T18:53:15Z

presto-product-tests/conf/docker/singlenode-postgresql/docker-compose.yml

+version: '2'
+services:
+
+  postgresql:


automation fails with: Caused by: java.net.UnknownHostException: postgres

It works fine locally. On Travis for some reason hostname comes after servicename, and a hostname field is ignored. This is weird. For now i'm just going to revert back the service name.

This allows us to avoid spinning up all backends (hadoop, mysql, cassandra, posgree) alltogether to save some memory

arhimondr · 2018-04-18T19:33:01Z

Can you also make Presto to use INFO for logging?

You mean setting all here to INFO at least?

https://github.com/prestodb/presto/blob/master/presto-product-tests/conf/presto/etc/log.properties#L10

findepi · 2018-04-18T19:42:33Z

@arhimondr you asked me for review, but i think @electrum's and @kokosing's is more than enough. Let me know if I'm still needed here.

kokosing · 2018-04-18T21:08:26Z

.travis.yml

@@ -73,7 +73,7 @@ script:
  - |
    if [[ -v PRODUCT_TESTS_BASIC_ENVIRONMENT ]]; then
      presto-product-tests/bin/run_on_docker.sh \
-        multinode -x quarantine,big_query,storage_formats,profile_specific_tests,tpcds
+        multinode -x quarantine,big_query,storage_formats,profile_specific_tests,tpcds,cassandra,mysql_connector,postgresql_connector,mysql


please file a ticket for that, to not to forget about it

kokosing · 2018-04-18T21:09:23Z

presto-product-tests/bin/run_on_docker.sh

  EXTERNAL_SERVICES="hadoop-master sqlserver"
 elif [[ "$ENVIRONMENT" == "singlenode-ldap" ]]; then
  EXTERNAL_SERVICES="hadoop-master ldapserver"
+elif [[ "$ENVIRONMENT" == "singlenode-mysql" ]]; then
+  EXTERNAL_SERVICES="hadoop-master mysql"


Can you file a ticket for tempto for that? I will try to fix that.

nezihyigitbasi · 2018-04-18T21:33:29Z

Thanks for fixing this @arhimondr, merged.

arhimondr requested review from findepi and kokosing April 18, 2018 18:04

facebook-github-bot added the CLA Signed label Apr 18, 2018

electrum reviewed Apr 18, 2018

View reviewed changes

kokosing suggested changes Apr 18, 2018

View reviewed changes

Run tests for different connectors in different builds

e5c5a79

This allows us to avoid spinning up all backends (hadoop, mysql, cassandra, posgree) alltogether to save some memory

arhimondr force-pushed the segregate-product-tests-by-connector-tested branch from f93d6c8 to e5c5a79 Compare April 18, 2018 19:14

findepi removed their request for review April 18, 2018 19:41

kokosing approved these changes Apr 18, 2018

View reviewed changes

nezihyigitbasi closed this Apr 18, 2018

arhimondr deleted the segregate-product-tests-by-connector-tested branch April 18, 2018 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run tests for different connectors in different builds #10440

Run tests for different connectors in different builds #10440

arhimondr commented Apr 18, 2018 •

edited

Loading

electrum left a comment

kokosing left a comment

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

kokosing Apr 18, 2018

arhimondr Apr 18, 2018

arhimondr commented Apr 18, 2018

findepi commented Apr 18, 2018

kokosing Apr 18, 2018

kokosing Apr 18, 2018

nezihyigitbasi commented Apr 18, 2018

Run tests for different connectors in different builds #10440

Run tests for different connectors in different builds #10440

Conversation

arhimondr commented Apr 18, 2018 • edited Loading

electrum left a comment

Choose a reason for hiding this comment

kokosing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arhimondr commented Apr 18, 2018

findepi commented Apr 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nezihyigitbasi commented Apr 18, 2018

arhimondr commented Apr 18, 2018 •

edited

Loading