Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get OTA running in docker compose #1175

Closed
michielbdejong opened this issue Sep 16, 2024 · 8 comments
Closed

Get OTA running in docker compose #1175

michielbdejong opened this issue Sep 16, 2024 · 8 comments

Comments

@michielbdejong
Copy link
Member

On my machine I can do:

michiel-2:edit.tosdr.org michiel$ docker ps -a
CONTAINER ID   IMAGE                             COMMAND                  CREATED         STATUS                        PORTS     NAMES
01b32f844d25   d53961c856d4                      "/bin/sh -c '. $NVM_…"   6 minutes ago   Exited (1) 6 minutes ago                edittosdrorg-ota-1
1c0fa2d65139   ota                               "/bin/sh -c '. $NVM_…"   3 days ago      Exited (255) 11 minutes ago             elastic_engelbart
bd42ff74b366   6bd90a35d7fc                      "/bin/sh -c '. $NVM_…"   3 days ago      Exited (1) 3 days ago                   clever_hawking
9a351658b210   1ec693117994                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   optimistic_roentgen
133ec2dfaa1c   1ec693117994                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   wonderful_hugle
22f0b2aaf3f8   2f8c38c8d75d                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   nice_einstein
b4addc33fb4b   74961f58b71e                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   exciting_torvalds
399e5ab6acb8   74961f58b71e                      "/bin/sh -c 'node se…"   3 days ago      Exited (1) 3 days ago                   flamboyant_spence
bf6002fa0831   46f48478527e                      "/bin/sh -c 'node se…"   3 days ago      Exited (1) 3 days ago                   infallible_ride
a367016ce2b0   b02aa118fb0d                      "/bin/sh -c 'node se…"   3 days ago      Exited (1) 3 days ago                   nervous_morse
5bef5dce3d51   5063bf567f3b                      "/bin/sh -c 'node se…"   3 days ago      Exited (127) 3 days ago                 zealous_goodall
de264f0bf91f   1525b244be1d                      "/bin/bash"              3 days ago      Exited (0) 3 days ago                   heuristic_mestorf
4bd74eda885f   edittosdrorg-web                  "bash docker-entrypo…"   3 days ago      Exited (137) 2 seconds ago              edittosdrorg-web-1
c53bc33e2d93   hypothesis/elasticsearch:latest   "/usr/local/bin/dock…"   3 days ago      Exited (143) 2 seconds ago              edittosdrorg-elasticsearch-1
44f4a00bf1db   postgres:11.5-alpine              "docker-entrypoint.s…"   3 days ago      Exited (0) 3 seconds ago                db
7ea7aad8572a   adminer                           "entrypoint.sh php -…"   3 days ago      Exited (0) 3 seconds ago                edittosdrorg-adminer-1
michiel-2:edit.tosdr.org michiel$ docker rm `docker ps -qa`
01b32f844d25
1c0fa2d65139
bd42ff74b366
9a351658b210
133ec2dfaa1c
22f0b2aaf3f8
b4addc33fb4b
399e5ab6acb8
bf6002fa0831
a367016ce2b0
5bef5dce3d51
de264f0bf91f
4bd74eda885f
c53bc33e2d93
44f4a00bf1db
7ea7aad8572a
michiel-2:edit.tosdr.org michiel$ docker pull pondersource/tosdr-ota
Using default tag: latest
latest: Pulling from pondersource/tosdr-ota
Digest: sha256:7b383c068355a823b3a22dcdabd5fd066d250423eeceb4c5e55137170042d143
Status: Image is up to date for pondersource/tosdr-ota:latest
docker.io/pondersource/tosdr-ota:latest

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview pondersource/tosdr-ota
michiel-2:edit.tosdr.org michiel$ docker compose up
[+] Running 5/5
 ✔ Container edittosdrorg-adminer-1        Created                                                                                                                                                                                             0.6s 
 ✔ Container edittosdrorg-web-1            Created                                                                                                                                                                                             0.6s 
 ✔ Container edittosdrorg-ota-1            Created                                                                                                                                                                                             0.6s 
 ✔ Container edittosdrorg-elasticsearch-1  Created                                                                                                                                                                                             0.6s 
 ✔ Container db                            Created                                                                                                                                                                                             0.6s 
Attaching to db, adminer-1, elasticsearch-1, ota-1, web-1
adminer-1        | [Mon Sep 16 08:41:12 2024] PHP 7.4.33 Development Server (http://[::]:8080) started
db               | 2024-09-16 08:41:12.231 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
db               | 2024-09-16 08:41:12.231 UTC [1] LOG:  listening on IPv6 address "::", port 5432
[...]

And then in a second terminal window:

michiel-2:edit.tosdr.org michiel$ docker ps
CONTAINER ID   IMAGE                             COMMAND                  CREATED          STATUS                    PORTS                                NAMES
3a81e20c4924   pondersource/tosdr-ota:latest     "/bin/sh -c '. $NVM_…"   12 seconds ago   Up 11 seconds             127.0.0.1:3000->3000/tcp             edittosdrorg-ota-1
4bd74eda885f   edittosdrorg-web                  "bash docker-entrypo…"   3 days ago       Up 11 seconds             0.0.0.0:9090->3000/tcp               edittosdrorg-web-1
c53bc33e2d93   hypothesis/elasticsearch:latest   "/usr/local/bin/dock…"   3 days ago       Up 11 seconds             127.0.0.1:9200->9200/tcp, 9300/tcp   edittosdrorg-elasticsearch-1
44f4a00bf1db   postgres:11.5-alpine              "docker-entrypoint.s…"   3 days ago       Up 11 seconds (healthy)   127.0.0.1:5432->5432/tcp             db
7ea7aad8572a   adminer                           "entrypoint.sh php -…"   3 days ago       Up 11 seconds             0.0.0.0:1337->8080/tcp               edittosdrorg-adminer-1
michiel-2:edit.tosdr.org michiel$ docker exec -it edittosdrorg-web-1 curl -d'{"fetch":"http://example.com","select":"body"}' http://ota:3000
{"fetch":"http://example.com","select":"body"}"<h1 id=\"exampledomain\">Example Domain</h1>\n<p>This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.</p>\n<p><a href=\"https://www.iana.org/domains/example\">More information…</a></p>"

@madoleary is this working for you too?

I just did a docker push so I think a docker pull pondersource/tosdr-ota should give you:

latest: digest: sha256:7b383c068355a823b3a22dcdabd5fd066d250423eeceb4c5e55137170042d143 size: 7836

and then it should work!
@madoleary
Copy link
Contributor

I pulled the image with the matching hash, but I still have the same error.

When I run: docker exec -it edittosdrorg_web_1 curl -d'{"fetch":"http://example.com","select":"body"}' http://ota:3000

I get: curl: (18) transfer closed with outstanding read data remaining

And the output of docker logs edittosdrorg_ota_1 is:

Node.js v20.17.0
listening on port 3000
{ fetch: 'http://example.com', select: 'body' }
/home/crawler/engine/node_modules/@puppeteer/browsers/lib/cjs/launch.js:310
                reject(new Error([
                       ^

Error: Failed to launch the browser process!
[263:263:0916/102307.128939:FATAL:zygote_host_impl_linux.cc(126)] No usable sandbox! Update your kernel or see https://chromium.googlesource.com/chromium/src/+/main/docs/linux/suid_sandbox_development.md for more information on developing with the SUID sandbox. If you want to live dangerously and need an immediate workaround, you can try using --no-sandbox.


TROUBLESHOOTING: https://pptr.dev/troubleshooting

    at ChildProcess.onClose (/home/crawler/engine/node_modules/@puppeteer/browsers/lib/cjs/launch.js:310:24)
    at ChildProcess.emit (node:events:531:35)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)

Node.js v20.17.0

@michielbdejong
Copy link
Member Author

which commit of edit.tosdr.org are you on? any uncommitted changes?
did you also remove the old containers? you need to make sure docker ps -a is empty before running docker compose up otherwise it will just start up the old ones

@michielbdejong
Copy link
Member Author

michielbdejong commented Sep 16, 2024 •
edited
Loading

you can run docker stop `docker ps -q` to stop all running containers (should not be necessary after docker compose down) and then docker rm `docker ps -qa` to remove them so they will be created fresh in your next docker compose up. This is one of the annoying things in docker compose which is why I don't really like it :)

@madoleary
Copy link
Contributor

I'm on commit:

commit 90edfb47a1ec845b73b621091bbe15613e6e1ab1 (HEAD -> ota-engine-integration, origin/ota-engine-integration)
Author: Michiel de Jong <[email protected]>
Date:   Fri Sep 13 13:59:08 2024 +0200

    Add ota server

I followed your steps and I still have the same error.

@michielbdejong
Copy link
Member Author

Hm, weird! Maybe it will go away.
Once I get the production server ready you'll be able to use that.

@michielbdejong
Copy link
Member Author

I pushed the ota image version 1.1, you should now see:

docker exec -it edittosdrorg-web-1 curl -d'{"fetch":"http://example.com","select":"body"}' http://ota:3000
Please send an Authorization header with the API_SECRET from this script's run environment

@michielbdejong
Copy link
Member Author

I pushed the ota image version 1.2, you should now see:

$ docker exec -it edittosdrorg-web-1 curl -d'{"fetch":"http://example.com","select":"body"}' -H 'Authorization: Bearer foo' http://ota:3000
{"fetch":"http://example.com","select":"body"}"<h1 id=\"exampledomain\">Example Domain</h1>\n<p>This domain is for 
[...]

@michielbdejong
Copy link
Member Author

OK we both saw this fail in gitpod, but it seems to work in Docker Desktop on Mac. Su.per.weird. Maybe the sandboxing is different if the host system is on ubuntu?

michielbdejong added a commit that referenced this issue Sep 16, 2024
@michielbdejong
Ref https://pptr.dev/guides/docker#usage, fix #1175
2f6d059
madoleary added a commit that referenced this issue Nov 5, 2024
@madoleary @michielbdejong
Validate ota crawler (#1187)
0d43307 
* starts implementation of engine

* working implementation of request to node server

* includes net::http implementation if preferred

* implementation of node server in app - not working

* disable Rack Attack

* seed DocumentTypes

* schema changes in dev DO NOT MERGE

* Document how to see document_types

* Add ota server

* image: pondersource/tosdr-ota:1.0

* ota image v1.1

* Use tosdr-ota:1.2

* Ref https://pptr.dev/guides/docker#usage, fix #1175

* Use pondersource/tosdr-ota:1.3

* manages ota env variables

* makes flash errors more robust

* implements ota crawler functionality, error handling for relevant methods

* updates views to account for new ota crawler

* removes outdated references to old crawlers

* removes ota engine dependency -- no longer needed!

* removes useless code

* removes yarn dependencies for ota engine

* clarifies standard error

* resolves conflict

* updates postgres to 14 to bring in line with produciton

* upgrades postgres in dev

* configures database for proper simulation of production env

* implements way to restore elasticsearch if documents missing for annotations

* adds last crawl date to documents

* adds es volumes

* refactor

* handles missing points after new crawl

* validates and converts selector

* implements way to restore annotations in es

* comments out useless code

---------

Co-authored-by: Michiel de Jong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@michielbdejong @madoleary