tests/integration.sh failing on outputs directory after a successful install_bootstrap run

[brainstorm]

ubuntu@ip-172-31-21-238:~/dcc-ops/test$ ./integration.sh
[2017-07-28 08:51:19.342][INFO  ][main:16 ] deleted old outputs
[2017-07-28 08:51:19.651][INFO  ][main:19 ] generated testing access token: 19947319-ce33-45b2-8b7c-bc556a3158dd
[2017-07-28 08:51:19.654][INFO  ][main:22 ] got redwood endpoint from dcc-ops/redwood/.env: http://dcc.umccr.nopcode.org/
[2017-07-28 08:51:19.657][INFO  ][main:24 ] doing upload with /home/ubuntu/dcc-ops/test/manifest.tsv
2017-07-28 08:51:20,205 INFO main:884 validated all metadata objects for output
2017-07-28 08:51:20,205 INFO main:890 number of metadata files written: 2
2017-07-28 08:51:20,206 INFO main:899 Uploading files.
2017-07-28 08:51:20,206 INFO main:900 NOTE: If it hangs IP may be blocked
2017-07-28 08:51:20,206 INFO main:950 counts	{"bundlesFound": 1}
2017-07-28 08:51:20,206 INFO main:950 counts	{"bundlesFound": 2}
2017-07-28 08:51:20,207 INFO register_upload:536 registering upload redwood metadata: dcc-metadata-client -m /outputs/registration.tsv -o /outputs/redwoodUploadManifest
2017-07-28 08:51:23,675 ERROR register_upload:543 error registering upload with redwood-metadata-server
2017-07-28 08:51:23,675 ERROR main:962 upload registration failed
cat: outputs/receipt.tsv: No such file or directory
cat: outputs/receipt.tsv: No such file or directory
[2017-07-28 08:51:23.926][INFO  ][main:31 ] the uploaded bundle with id  has metadata.json with object id
[2017-07-28 08:51:23.928][INFO  ][main:33 ] downloading metadata.json with object id
ERROR: Bad parameter(s): Invalid option: --object-id: /outputs/test_download is not a valid UUID

[brainstorm]

Another (perhaps related?) issue I'm having after a successful install_bootstrap run is a functioning portal:

http://dcc.umccr.nopcode.org

But not reachable via https, therefore the LOGIN page (among others that use https only) cannot be reached:

https://dcc.umccr.nopcode.org

I've checked the security groups on AWS and the firewall in the instance and it's neither since I can see via tcpdump -vv -i eth0 port 443 that requests/packets arrive at the instance without much trouble. It seems to be a problem with nginx/letsencrypt perhaps?:

https://stackoverflow.com/questions/36725901/connection-refused-nginx-https-reverse-proxy-in-docker-container

The containers seem to be exposing port 443 correctly though for several docker instances:

CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS                          PORTS                                                                                        NAMES
54d5606f8cf0        quay.io/ucsc_cgl/action-service:1.0.5                 "/home/ubuntu/setu..."   5 hours ago         Up 4 hours                      0.0.0.0:32778->8082/tcp                                                                      action-service
6a1c0e5a8fc0        boardwalk_nginx                                       "/bin/sh -c ./star..."   5 hours ago         Up 4 hours                      0.0.0.0:32782->80/tcp, 0.0.0.0:32781->443/tcp                                                boardwalk_nginx_1
f0f7f2aa1ae3        quay.io/ucsc_cgl/dashboard:1.0.7                      "/usr/bin/supervisord"   5 hours ago         Up 4 hours                      0.0.0.0:32776->80/tcp, 0.0.0.0:32775->443/tcp                                                boardwalk_dcc-dashboard_1
b937a573c872        quay.io/ucsc_cgl/dashboard-service:1.0.3              "/usr/bin/supervisord"   5 hours ago         Up 4 hours                      80/tcp, 443/tcp, 0.0.0.0:32772->9200/tcp                                                     boardwalk_dcc-dashboard-service_1
9c0d30ee3322        quay.io/ucsc_cgl/metadata-indexer:2.0.5               "./run2.sh --stora..."   5 hours ago         Up 4 hours                      0.0.0.0:32773->9200/tcp                                                                      boardwalk_dcc-metadata-indexer_1
413b7a29ebeb        postgres:9.6-alpine                                   "docker-entrypoint..."   5 hours ago         Up 4 hours                      5432/tcp                                                                                     boardwalk_db_1
13a0c0e7ead2        postgres:9.6-alpine                                   "docker-entrypoint..."   5 hours ago         Up 4 hours                      0.0.0.0:32768->5432/tcp                                                                      bd-chart
8cc73b57e54a        postgres:9.6-alpine                                   "docker-entrypoint..."   5 hours ago         Up 4 hours                      5432/tcp                                                                                     login-db
63652def47e9        docker.elastic.co/elasticsearch/elasticsearch:5.2.2   "/bin/bash bin/es-..."   5 hours ago         Up 4 hours                      9300/tcp, 0.0.0.0:32774->9200/tcp                                                            elasticsearch1
996bb9695417        quay.io/ucsc_cgl/boardwalk:1.0.0                      "./run.sh"               5 hours ago         Up 4 hours                      0.0.0.0:32771->80/tcp, 0.0.0.0:32770->443/tcp, 0.0.0.0:32769->3000/tcp                       boardwalk_boardwalk_1
e033d38c5fbc        postgres:9.6-alpine                                   "docker-entrypoint..."   5 hours ago         Up 4 hours                      0.0.0.0:32777->5432/tcp                                                                      boardwalk-billing
5308264e472e        docker.elastic.co/elasticsearch/elasticsearch:5.2.2   "/bin/bash bin/es-..."   5 hours ago         Up 4 hours                      9200/tcp, 9300/tcp                                                                           boardwalk_elasticsearch2_1
7b6b2b29e545        jrcs/letsencrypt-nginx-proxy-companion:v1.5           "/bin/bash /app/en..."   5 hours ago         Up 4 hours                                                                                                                   core-letsencrypt-nginx-companion
106835c90604        jwilder/docker-gen:0.7.3                              "/usr/local/bin/do..."   5 hours ago         Up 4 hours                                                                                                                   core-config-gen
da4fb26cc4b0        nginx:1.11.10                                         "nginx -g 'daemon ..."   5 hours ago         Up 4 hours                      0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                     core-nginx
0b8710c019d0        quay.io/ucsc_cgl/redwood-storage-server:r1.0.26       "/bin/sh -c 'dcc-s..."   5 hours ago         Up 4 hours                      5431/tcp                                                                                     redwood-storage-server
f940beb2f4a8        quay.io/ucsc_cgl/redwood-metadata-server:r0.0.24      "/bin/sh -c 'dcc-m..."   5 hours ago         Up 4 hours                      8444/tcp, 8544/tcp                                                                           redwood-metadata-server
1f2e6c5054a3        quay.io/ucsc_cgl/redwood-auth-server:r4.3.4           "/bin/sh -c 'dcc-a..."   5 hours ago         Up 4 hours                      8443/tcp, 8543/tcp                                                                           redwood-auth-server
85859eb5e2e7        schickling/postgres-backup-s3                         "sh run.sh"              5 hours ago         Up 4 hours                                                                                                                   redwood-auth-backup
e34f2588a03a        agmangas/mongo-backup-s3                              "python -u /app/ru..."   5 hours ago         Up 4 hours                                                                                                                   redwood-metadata-backup
20b3a0c87f9c        postgres:9.6.2                                        "docker-entrypoint..."   5 hours ago         Up 4 hours                      5432/tcp                                                                                     redwood-auth-db
3272e48be5fd        mongo:3.4.2                                           "docker-entrypoint..."   5 hours ago         Up 4 hours                      27017/tcp                                                                                    redwood-metadata-db
53c774ddd6ef        consonance_client                                     "dockerize -wait t..."   5 hours ago         Restarting (0) 13 seconds ago                                                                                                consonance_client_1
b7d845d84346        consonance_provisioner                                "dockerize -wait t..."   5 hours ago         Up 4 hours                                                                                                                   consonance_provisioner_1
c01cfa2e157f        consonance_coordinator                                "dockerize -wait t..."   5 hours ago         Up 4 hours                                                                                                                   consonance_coordinator_1
24422a729e3f        consonance_webservice                                 "dockerize -wait t..."   5 hours ago         Up 4 hours                      0.0.0.0:8080-8081->8080-8081/tcp                                                             consonance-webservice
d483dcd573d2        postgres:9.5                                          "docker-entrypoint..."   5 hours ago         Up 4 hours                      5432/tcp                                                                                     consonance_postgres_1
26874e7278e0        rabbitmq:3.5.6-management                             "/docker-entrypoin..."   5 hours ago         Up 4 hours                      4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp   consonance_rabbitmq_1

Any ideas on what could be causing this, @briandoconnor ?

[caaespin]

If you check under dcc-ops/common/certs/, do you see the certificates?

[brainstorm]

Thanks for the hint @caaespin! I think I just found the crux of the matter:

ubuntu@ip-172-31-21-238:~/dcc-ops/common/certs$ ls
auth.dcc.umccr.nopcode.org  auth.http:  dcc.umccr.nopcode.org  dhparam.pem  http:  metadata.dcc.umccr.nopcode.org  metadata.http:  storage.dcc.umccr.nopcode.org  storage.http:

Again, input filtering during the install_bootstrap script, I guess (see files ending in http:). Related with issue #103.

[brainstorm]

@caaespin Which container/script would allow me to recreate that certs directory and its certificates?

[caaespin]

@brainstorm , you could try removing (I would mv them somewhere just in case) the contents of the certs folder and running again the bootstrap script. As you pointed out in issue #106 , it should be "Domain Name", not "Base URL". If the protocol is included when feeding the answers to the bootstrap script, it will break or cause unexpected things downstream.

So I would:

• do sudo docker-compose -f base.yml -f prod.yml down -v under dcc-ops/common
• Delete (or move somewhere else) the contents of the dcc-ops/common/certs/ folder
• Run the bootstrap script and just run the commons part ( i.e. Are you ready to launch the public-facing gateway nginx server?)

Give that a try and see if that fixes the https issue.

[brainstorm]

Yes it did! Thanks much, closing 👍

Also, integration.sh tests passing, so fixing/clarifying the install_bootstrap URL step should spare much headache to other users ;)