Docker build fails #91
Comments
|
it seems that you need to explicitely call for the mysql debian image, or change the package manager from apt-get to yum. The simplest is likely to change in vdb_docker.df the line:
to
|
|
ok the build process seems to be ok, but launch fails. I followed the following steps to recreate the containers and update the code: added email and NCBI API KEY in docker.ini, then I had to slightly modify vdb_docker.df becomes and becomes then: Then rest_variantvalidator_restvv_1 fails to launch. I retried with the following end sequence and code from rest_variantValidator release 2.2.0 instead of just cloning the repo or pulling new code, but same error at startup: Any help would be highly appreciated. |
|
I'm no expert on Docker (actually, I rarely use it), but I remember changes have been made recently about whether or not VV had to use pooling of database connections. Remember those race conditions that we saw? The error is about this file. It seems that perhaps UTA is confused about whether to use pooling or not. Do you think you have write access to |
|
Thx @ifokkema, however, the code on this file looks good to me, as far as I can tell. In def connect(), if pooling is enabled, the object is correctly instanciated: then self._pool should be accessible when closing: Then I'm not sure of what is wrong here. Or maybe the pooling attribute has changed after the first connection, from False to True? That would be weird. And the code is 2 yaers old. Well... |
|
I didn't mean that the bug is necessarily in this file; it might as well be in its usage outside of this file. I don't see |
|
well, you're probably right, as usual, but in my opinion just checking could lead to improper closing of the connection, or worse, to leave it open which would quickly result in a freeze of the whole system due to a lack of available connections. I'll maybe make some tests tomorrow. thanks anyway. |
|
Well, the connection, if it exists, will still be closed, of course. The Anyway, in the worst-case scenario, the connection should close when the Python program ends. |
|
ok I'll give it a try asap with a try/except block
But I wouldn't bet on it. From my experience, depending on the way you handle psycopg2 connections, the syntax differs then if a connection is opened with a connexion pool you must release it the same way - then the else may fail as well. But it may work if the self.pooling boolean has been modify in the meantime, right.
again, not sure about that, even if it seems good sense. I've had the case in a flask app where connections were not correctly closed, and when ran as a wsgi app, connections remained opened, and eventually led to the system freeze. Apache kind of keeps the scripts in memory, to be able to execute them quicker (such as in mod_perl, for example). You probably know better on this. |
That may actually lead to the connection not being closed, as the
Sure, but there's nothing to release as there is no
That's my guess, indeed, but only a guess. It may also be that the connection has already closed and
Well, I hardly use WSGI, so I wouldn't know better, actually... I just quickly googled my assumption that the connections would be closed, the first link mentioned that indeed this would happen, and I didn't look further. So you might be right there! Either way, it's worth a shot, as now, nothing is working. |
|
@beboche @ifokkema I have updated the docker with mysql:5.7-debian
I'll see if I can build the container at my end |
|
Nice I'll give it a try asap |
|
One sec. forgot to add it to the API. Doing now |
|
Ok I have a bunch of meetings today, anyway
Le 19 avr. 2023 à 10:06, Peter J. Freeman ***@***.***> a écrit :
One sec. forgot to add it to the API. Doing now
—
Reply to this email directly, view it on GitHub<#91 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AG6LYSKHUB6D2X5DT43DUC3XB6MJHANCNFSM6AAAAAAVSLN2H4>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@Peter-J-Freeman is it ok from your side? |
|
I ran it this afternoon and everything installed. |
|
fails for me when docker-compose up (last step of update) - followed the Docker.md instructions, last step: seems to come from me but the server looks fine |
|
It's a debian 11 VM - when I launch back the last snapshot, everything runs fine with: VariantValidator version 2.1.1.dev2+g294fd63 |
|
So you are seeing errors but everything runs? |
|
no sorry. I've got a VM running legacy VV. I want to update. |
|
I don't know what's going on. I then exactly followed the docker.md instructions. Everything's fine until the second last step (building vdb), which seems to somewhat fail: Then if I try the last command, this is the restvv container that fails the same than on my previous machine: Note: I am able to run again quite successfully
(no error), but
fails the same: What would you suggest? I'll have a look at the manual install (outside docker), but I'm afraid this will be a pain in the ass. |
|
looks like restvv does not wait for vdb to be up to launch? should it? |
|
No. Interesting point thought. I'm going to see if I can tweak the start up order |
|
Try replacing the content of docker-compose with Once you complete the docker-compose build step run $ docker-compose up
# Shut down when database ready to accept commections with ctrl c
$ docker-compose up |
|
Containers should now start in order from top to bottom with vvrest last |
|
No - restvv is still starting before vdb is done: below the 2nd There must be a trick - I'll search for it tomorrow morning. |
|
Did you try to do docker-compose run again. Wondered if the API is sparking up |
|
yes, I tried several times - the startup sequence is identical The vdb container takes a while (1-2 minutes) to show up as you can see on the log |
|
Hi, good and bad news: I modified the docker-compose.yml file to take advantage of the healthcheck capability of docker. and the vvta: modifications on the healthcheck attribute (do we say that for yml?) for the 1st and depends_on for the 2nd. but sadly for me, as you can see the restvv container still fails miserably. This time no clue. |
This error only happens, as far as I know, when |
|
Right, I'd need to fork variantValidator and point the Dockerfile to my repo. However I'm not familiar with the egg part of the command
in the Dockerfile. Does it require a specific action on the repo, apart from modifying the code of the vvMixinInit.py module file? I'll not be able to test until at least tuesday. Thx for the tip. |
weird this one is suppposed to be fixed. |
|
@hotsoupisgood , think you have not set the path to your SeqRepo database correctly. See here #96 (comment) |
yes those are made, and the seqrepo is there: |
|
Sorry, @beboche . Was getting confused between 2 issues. Back on this one |
|
The issue that crops up every time on your error logs @beboche is this one vvta_1 | PostgreSQL Database directory appears to contain a database; Skipping initialization The section I am doing a full system prune of docker on my system and re-building using the .yml file in master and the build process in here. https://github.com/openvar/rest_variantValidator/blob/master/docs/DOCKER.md. Current master version. Will get back to you. |
|
Also @beboche . Have you considered building locally. Since you update regularly it might be easier for you to maintain. Once built it is generally updating code and restarting or updating a database and re-starting. Happy to help you through the build. |
Ok, that worked for me. @beboche what do you get if you run docker-compose up vvta |
maybe I should, indeed
|
you do not have that message? |
Yes I did this time. So its not that. mine still starts. Just working through the log see if we can figure out exactly what fails. try docker-compose up vdb. Make sure that says it is ready to accept connections. I still think that you might be better setting up locally though. Save you headaches in the future. Are you on a linux server? The build isnt's particularly difficult. If you can install conda, postgres mysql and sqlite. it doesn't take long and once installed it is very reliable |
yes almalinux 9 - ok I'll consider that |
|
Let me know if you want to set up a call to go through it |
I'll check the doc and let you know, thx |
|
ran vdb again: |
|
so the databases seem to run just fine. But when you hit docker-compose up, things start to exit? |
|
well the databases start well but the vv worker fails to boot |
|
check the last lines: |
|
OK, so I think it must be this
It looks like all databases are fine but for some reason gunicorn is not happy. Seems to fail |
|
I get the following and everything keeps running
|
|
I think I have a clue I don't I'll try to run gunicorn in preload moed, to get more detailed error message. |
|
Nice. At least we now have a clue. |
|
Hi - I ran gunicorn in preload mode, this gave me the following python error: Note the line _mysql_connector.MySQLInterfaceError: Host '172.19.0.5' is not allowed to connect to this MySQL server I suppose 172.19.0.5 is the vv docker address? So it's not allowed but not sure why. I cut the log but the mysql vdb started fine |
|
This may be the difference. I'm building on a mac M1 With ARM using the container from here For Arm chips e.g. Apple M1FROM biarms/mysql:5.7 For other processors you need to select this. For Intel chipsFROM mysql:5.7-debianI was asked to change this to 5.7-debian bit I'm not sure if this is correct The documentation seem to say 5.7 should work. Try replacing the 5.7-debian with just 5.7 |
|
mysql:5.7 is an oracle linux, the package manager is yum and not aptitude - then the vdb_docker.df won't work. I'll run some tests we should be able to fix this. |
|
if I let we may be getting closer |
|
It may be related to mysql 5.7 with some architectures and memory consumption - I'm experiencing the 16Go consumption described here: I'm trying mysql 8 |
|
Finally the docker resttvv api is running. I just had to change the mysql version in the vdb_docker.df file. Working version for my config (AlmaLinux 9 on intel chips) is: at the beginning of the df file. I also had to change the Dockerfile very last line from: to to be able to reach VV API from another machine. |
|
Good work. I am making edits. Shall I pull these into the repo? |
I think so |
Hi,
was trying to update my VV docker - I deleted all the containers according to DOCKER.md to start from scratch.
I cloned the current vv repo, then added my NCBI key and then launched the build affter having pulled
docker-compose build --no-cacheand I got the following error:
Does that ring a bell to you?
thx
The text was updated successfully, but these errors were encountered: