Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup with external db not working (Composer errors) #155

Closed
Jan-NiklasB opened this issue Jun 22, 2021 · 18 comments
Closed

Setup with external db not working (Composer errors) #155

Jan-NiklasB opened this issue Jun 22, 2021 · 18 comments
Assignees
Labels
help wanted Extra attention is needed

Comments

@Jan-NiklasB
Copy link

Hey,
I just tried to setup openeats on my server running an MariaDB.
After reading the setup guide and trying around I always get errors on running quick-setup.py

My files:

docker-prod.yml

version: '2.3'
services:
  api:
    image: openeats/openeats-api
    command: /startup/prod-entrypoint.sh
    restart: on-failure
    volumes:
      - static-files:/code/static-files
      - site-media:/code/site-media
    depends_on:
      db:
        condition: service_healthy
    env_file:
      env_prod.list
  web:
    image: openeats/openeats-web
    command: yarn start
    depends_on:
      - api
    volumes:
      - public-ui:/code/build
    env_file:
      env_prod.list
  #db:
  #  image: mariadb:5.5.64
  #  volumes:
  #    - database:/var/lib/mysql
  #  healthcheck:
  #    test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
  #    timeout: 20s
  #    retries: 20
  #  env_file:
  #    env_prod.list
  nginx:
    image: openeats/openeats-nginx
    command: ./start.sh
    volumes:
      - static-files:/var/www/html/openeats-static/static-files
      - site-media:/var/www/html/openeats-static/site-media
      - public-ui:/var/www/html/openeats-static/public-ui
    depends_on:
      - api
      - web
    env_file:
      env_prod.list
volumes:
  #database:
  public-ui:
  static-files:
  site-media:

docker-prod.override.yml

version: '2.3'
services:
  nginx:
    ports:
      - "357:80"
    restart: always
  api:
    restart: always
  web:
    restart: on-failure
 # db:
 #   restart: always
 
env_prod.list

# Database config
MYSQL_DATABASE=openeats
MYSQL_USER=openeats
MYSQL_HOST=192.168.13.123
MYSQL_PORT=3306
MQSQL_ROOT_PASSWORD=Password

# Django config
API_URL=0.0.0.0:8000
API_PORT=8000
DJANGO_SECRET_KEY=***
DJANGO_SETTINGS_MODULE=base.settings
DJANGO_DEBUG=False
ALLOWED_HOST=eat.my-domain.de
HTTP_X_FORWARDED_PROTO=true

# Node config
NODE_ENV=production
NODE_URL=localhost:8080
NODE_API_URL=https://eat.my-domain.de
NODE_LOCALE=de

Error reported by quick-setup.py:

root@s02:/opt/OpenEats# python3 ./quick-start.py
==================
Downloading Images
==================
latest: Pulling from openeats/openeats-api
Digest: sha256:7d73944dc653a2781c566055c3ecd5c4d126766e256dc27dbcc36c37fae62b9a
Status: Image is up to date for openeats/openeats-api:latest
docker.io/openeats/openeats-api:latest
latest: Pulling from openeats/openeats-web
Digest: sha256:f49139cd0daa83a3420c7993c080e398c58de327032dbfaf26323ea1b4c17526
Status: Image is up to date for openeats/openeats-web:latest
docker.io/openeats/openeats-web:latest
latest: Pulling from openeats/openeats-nginx
Digest: sha256:d30864b7f37f6ad399d97f00e8f07fcef26256cf4c75facfab6f1c7517a1b117
Status: Image is up to date for openeats/openeats-nginx:latest
docker.io/openeats/openeats-nginx:latest
==================
Starting OpenEats
==================
Using remote DB...
ERROR: Service 'api' depends on service 'db' which is undefined.
ERROR: Service 'api' depends on service 'db' which is undefined.
ERROR: Service 'api' depends on service 'db' which is undefined.
ERROR: Service 'api' depends on service 'db' which is undefined.
App started. Please wait ~30 seconds for the containers to come online.

Can someone tell me what else I have to change?

Thanks in advance!

@rustymyers
Copy link
Member

In the docker-prod.yml you should remove the

    depends_on:
      db:
        condition: service_healthy

See the example here:
https://github.com/open-eats/OpenEats/blob/master/docs/samples/sample_docker_prod_remote_db.yml

@rustymyers rustymyers self-assigned this Jun 22, 2021
@rustymyers
Copy link
Member

Also, check this comment for recent suggestion on getting remote DB password to work:
#145 (comment)

@Jan-NiklasB
Copy link
Author

Jan-NiklasB commented Jun 22, 2021

@rustymyers
Thanks for the fast response.
I already found out thet the "depends on" has to be commented out myself.

For the password I'm still fiddeling around, as the docker compose command doesn't seeams to work for me even if I set both password variables....

my current env_prod.list

# Database config
MYSQL_DATABASE=openeats
MYSQL_USER=openeats
MYSQL_HOST=192.168.13.123
MYSQL_PORT=3306
MQSQL_ROOT_PASSWORD=PassWORD!
MYSQL_PASSWORD=PassWORD!

# Django config
API_URL=0.0.0.0:8000
API_PORT=8000
DJANGO_SECRET_KEY=****
DJANGO_SETTINGS_MODULE=base.settings
DJANGO_DEBUG=False
ALLOWED_HOST=eat.my-domain.de
HTTP_X_FORWARDED_PROTO=true

# Node config
NODE_ENV=production
NODE_URL=localhost:8080
NODE_API_URL=https://eat.my-domain.de
NODE_LOCALE=de

Console output:


root@s02:/opt/OpenEats# python3 ./quick-start.py
==================
Downloading Images
==================
latest: Pulling from openeats/openeats-api
Digest: sha256:7d73944dc653a2781c566055c3ecd5c4d126766e256dc27dbcc36c37fae62b9a
Status: Image is up to date for openeats/openeats-api:latest
docker.io/openeats/openeats-api:latest
latest: Pulling from openeats/openeats-web
Digest: sha256:f49139cd0daa83a3420c7993c080e398c58de327032dbfaf26323ea1b4c17526
Status: Image is up to date for openeats/openeats-web:latest
docker.io/openeats/openeats-web:latest
latest: Pulling from openeats/openeats-nginx
Digest: sha256:d30864b7f37f6ad399d97f00e8f07fcef26256cf4c75facfab6f1c7517a1b117
Status: Image is up to date for openeats/openeats-nginx:latest
docker.io/openeats/openeats-nginx:latest
==================
Starting OpenEats
==================
Using remote DB...
Taking a image backup save to 'site-media'...
Stopping openeats_nginx_1 ... done
Stopping openeats_api_1 ... done
Recreating openeats_api_1 ... done
Recreating openeats_web_1 ... done
Recreating openeats_nginx_1 ... done
App started. Please wait ~30 seconds for the containers to come online.
root@s02:/opt/OpenEats# mc

root@s02:/opt/OpenEats# docker-compose -f docker-prod.yml run --rm --entrypoint 'python manage.py createsuperuser' api
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'openeats'@'172.18.0.3' (using password: NO)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 59, in execute
    return super().execute(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 352, in execute
    self.check_migrations()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 442, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 212, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table
    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (1045, "Access denied for user 'openeats'@'172.18.0.3' (using password: NO)")

The DB is accessable, as I can connect via mysql on cmd without problems....

@rustymyers
Copy link
Member

Try cleaning everything out and starting fresh.
#147 (comment)
(I have not tried it and I am no expert)

@Jan-NiklasB
Copy link
Author

@rustymyers

Sadly no luck:
(don't wonder, I already was deleting als containers and volumes)

root@s02:/opt/OpenEats# docker stop openeats_web_1 openeats_nginx_1 openeats_api_1 openeats_db_1 ; docker rm openeats_web_1 openeats_nginx_1 openeats_api_1 openeats_db_1 ; docker volume rm openeats_database
openeats_web_1
Error response from daemon: No such container: openeats_nginx_1
Error response from daemon: No such container: openeats_api_1
Error response from daemon: No such container: openeats_db_1
openeats_web_1
Error: No such container: openeats_nginx_1
Error: No such container: openeats_api_1
Error: No such container: openeats_db_1
Error: No such volume: openeats_database
root@s02:/opt/OpenEats# python3 ./quick-start.py
==================
Downloading Images
==================
latest: Pulling from openeats/openeats-api
Digest: sha256:7d73944dc653a2781c566055c3ecd5c4d126766e256dc27dbcc36c37fae62b9a
Status: Image is up to date for openeats/openeats-api:latest
docker.io/openeats/openeats-api:latest
latest: Pulling from openeats/openeats-web
Digest: sha256:f49139cd0daa83a3420c7993c080e398c58de327032dbfaf26323ea1b4c17526
Status: Image is up to date for openeats/openeats-web:latest
docker.io/openeats/openeats-web:latest
latest: Pulling from openeats/openeats-nginx
Digest: sha256:d30864b7f37f6ad399d97f00e8f07fcef26256cf4c75facfab6f1c7517a1b117
Status: Image is up to date for openeats/openeats-nginx:latest
docker.io/openeats/openeats-nginx:latest
==================
Starting OpenEats
==================
Using remote DB...
Creating volume "openeats_static-files" with default driver
Creating volume "openeats_site-media" with default driver
Creating openeats_api_1 ... done
Creating openeats_web_1 ... done
Creating openeats_nginx_1 ... done
App started. Please wait ~30 seconds for the containers to come online.
root@s02:/opt/OpenEats# docker-compose -f docker-prod.yml run --rm --entrypoint 'python manage.py createsuperuser' api
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
_mysql_exceptions.OperationalError: (1045, "Access denied for user 'openeats'@'172.18.0.3' (using password: NO)")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 316, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.6/site-packages/django/contrib/auth/management/commands/createsuperuser.py", line 59, in execute
    return super().execute(*args, **options)
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 352, in execute
    self.check_migrations()
  File "/usr/local/lib/python3.6/site-packages/django/core/management/base.py", line 442, in check_migrations
    executor = MigrationExecutor(connections[DEFAULT_DB_ALIAS])
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 18, in __init__
    self.loader = MigrationLoader(self.connection)
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 49, in __init__
    self.build_graph()
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/loader.py", line 212, in build_graph
    self.applied_migrations = recorder.applied_migrations()
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 61, in applied_migrations
    if self.has_table():
  File "/usr/local/lib/python3.6/site-packages/django/db/migrations/recorder.py", line 44, in has_table
    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 255, in cursor
    return self._cursor()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 232, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 227, in get_new_connection
    return Database.connect(**conn_params)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/__init__.py", line 86, in Connect
    return Connection(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/MySQLdb/connections.py", line 204, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
django.db.utils.OperationalError: (1045, "Access denied for user 'openeats'@'172.18.0.3' (using password: NO)")

@Jan-NiklasB
Copy link
Author

For completenes I just changed my db user to no password, and everything works fine, except for entering the Django-backend (CSRF check failed)....

So seems like there's still a bit work to do.....

@rustymyers
Copy link
Member

Try MARIADB_ROOT_PASSWORD?

Source: https://hub.docker.com/_/mariadb/

@Jan-NiklasB
Copy link
Author

Jan-NiklasB commented Jun 22, 2021

Try MARIADB_ROOT_PASSWORD?

Source: https://hub.docker.com/_/mariadb/

Tried this, but doesn't work either:

# Database config
MYSQL_DATABASE=openeats
MYSQL_USER=openeats
MYSQL_HOST=192.168.13.123
MYSQL_PORT=3306
MQSQL_ROOT_PASSWORD=eatME
MYSQL_PASSWORD=eatME
MARIADB_ROOT_PASSWORD=eatME
MARIADB_PASSWORD=eatME

For some reason openeats seems to don't use the password supplied....

Regarding the CSRF error, was my fault, if anyone is running into this in the future, try the option "ProxyPreserveHost On" in your revese proxy.

@rustymyers
Copy link
Member

It seems the code is looking for os.environ.get('MYSQL_ROOT_PASSWORD', '') to get the password for mysql. I wonder if it can't find it, does it default to No password or empty password, or are those the same thing to mysql?

Is your password being changed by the shell? Do you have an ! or * (other special characters)? Would that effect the mysql_root_password?

@rustymyers rustymyers added the help wanted Extra attention is needed label Jun 22, 2021
@Jan-NiklasB
Copy link
Author

It seems the code is looking for os.environ.get('MYSQL_ROOT_PASSWORD', '') to get the password for mysql. I wonder if it can't find it, does it default to No password or empty password, or are those the same thing to mysql?

Is your password being changed by the shell? Do you have an ! or * (other special characters)? Would that effect the mysql_root_password?

If "No Password" is equal to "empty password" depends on the constraints set in the table (as far as my smal knowledg reaches).

I already tried passwords with and without special chars (in my case "!"), so it doesn't seeams to affect the behavior... maybe the order of the variables is important to the parser?

@rustymyers
Copy link
Member

I created a new instance with a custom MYSQL_ROOT_PASSWORD:

# Database config
MYSQL_DATABASE=openeats
MYSQL_ROOT_PASSWORD=rootPoop

This seems to work as expected and the database is created, as is the superuser. I don't know that this means much, but the error for connecting is still showing in my docker db container:

2021-06-22 16:42:03 57 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
2021-06-22 16:42:33 58 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
2021-06-22 16:43:03 59 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

The web services are working, I can create a new recipe and log into the admin interface. I wonder if the issue isn't the database? #61 (comment)

@Jan-NiklasB
Copy link
Author

Jan-NiklasB commented Jun 22, 2021

So for better traceback my full setup:

OS:
Debain 11 Bullseye (testing) [for hardware compatibility]
Kernel 5.10.0-2amd64

Software:
Webmin 1.979
Apache 2.4.46
MariaDB 10.5.8
Docker 20.10.2

Setup:

Router
    |
Server Firewall
    |---------------------------------------|
Apache (name based)                      MariaDB
    |                                       |
Docker ------------------------------------ | (direct connection via local IP)
    |
Nginx (as part of openeats)
    |
Openeats services

I hope that maybe helps to localize the issue.

@Jan-NiklasB
Copy link
Author

Jan-NiklasB commented Jun 22, 2021

I created a new instance with a custom MYSQL_ROOT_PASSWORD:

# Database config
MYSQL_DATABASE=openeats
MYSQL_ROOT_PASSWORD=rootPoop

This seems to work as expected and the database is created, as is the superuser. I don't know that this means much, but the error for connecting is still showing in my docker db container:

2021-06-22 16:42:03 57 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
2021-06-22 16:42:33 58 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
2021-06-22 16:43:03 59 [Warning] Access denied for user 'root'@'localhost' (using password: NO)

The web services are working, I can create a new recipe and log into the admin interface. I wonder if the issue isn't the database? #61 (comment)

Well, the problem shows up when the external db hast to be populated with the initial data.
The internal DB is somehow populated ignoring the password problem....

Edit: Regarding the comment, I don't think so, as at the moment I'm running openeats with a db user without PW and everything works fine, no errors at all.
So I would guess it's a bug in the function to connect to the db.

@rustymyers
Copy link
Member

Have you updated the docker-compose.yml file to remove the references to db? I was able to set up a phpmyadmin/mysql instance on the same host and point it to the database.

At first I got _mysql_exceptions.OperationalError: (1049, "Unknown database 'oedatabase'") from the command docker logs openeats_api_1. Then I got _mysql_exceptions.ProgrammingError: (1146, "Table 'oedatabase.django_session' doesn't exist") and did the migration with the following commands:

docker exec -it openeats_api_1 /bin/sh
$ python manage.py makemigrations
$ python manage.py migrate
$ python manage.py createsuperuser

@Jan-NiklasB
Copy link
Author

Have you updated the docker-compose.yml file to remove the references to db? I was able to set up a phpmyadmin/mysql instance on the same host and point it to the database.

At first I got _mysql_exceptions.OperationalError: (1049, "Unknown database 'oedatabase'") from the command docker logs openeats_api_1. Then I got _mysql_exceptions.ProgrammingError: (1146, "Table 'oedatabase.django_session' doesn't exist") and did the migration with the following commands:

docker exec -it openeats_api_1 /bin/sh
$ python manage.py makemigrations
$ python manage.py migrate
$ python manage.py createsuperuser

Yes, i did.

In my log, the error is the same as posted above.

Any command including a call to the DB ended in "Access denied" until I changed the user to not require a authentication.

After this, I did the steps you mentioned with the migration, which worked as they should.

@rustymyers
Copy link
Member

Try updating your env_prod.list to use MYSQL_ROOT_PASSWORD instead of MQSQL_ROOT_PASSWORD

@Jan-NiklasB
Copy link
Author

Jan-NiklasB commented Jun 22, 2021

Try updating your env_prod.list to use MYSQL_ROOT_PASSWORD instead of MQSQL_ROOT_PASSWORD

Wow......

Sorry for wasting your time by this stupid mistake....

Everthing works now as exspected after applying the migrations after the restart again -.-

Thank you very much for your help!

@rustymyers
Copy link
Member

Sorry it took me so long to see that :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

2 participants