(Question) Debugging Flask + Celery in Docker Compose?

[ccrvlh]

Hey all. I was wondering if you guys could help shed some light debugging a project with a few containers.
I basically have:

• Flask App (running with app.ini uwsgi)
• Celery Worker
• Celery Beat (Scheduler)
• Redis
• RabbitMQ
• Flower

All in a docker-compose.yml file. Before moving the project to containers, I would just use the Debugger regularly with as a Flask App, which really helps during development, but can't seem to find a way to do that in my current setup.

I've seen this question and one of the answers says:

Attach to your application using an attach configuration in launch.json. This will depend on the language/framework you are using. Are you using Node, .NET Core, Python, others, a mix?

But I don't really understand how to do that. If I press F5 the "Add Docker" options pops up, but it makes a whole new requirements.txt, Dockerfile and docker-compose.yml file, I don't get it. Should. I don't get how that works? The containers are already up (right clicking my docker-compose.yml and 'Docker Compose Up` ).

I created a launch.json configuration with attatch request, like:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 8080
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/app"
                }
            ]
        }
    ]
}

After running the containers (docker-compose up -d) I went to the debugger and clicked "play", but nothing happens. I just wanted to run my code step by step with breakpoints and all. Ideally, I would like to be able to debug both Flask, and Celery tasks, so probably would want to attach de debugger to multiple processes?

Any ideas that could be of any help?
Thanks a lot

[bwateratmsft]

Hey @lowercase00, I'd suggest trying to blend what gets created by "Add Docker Files to Workspace..." with your solution. For example, in order to facilitate Python debugging along with the "Python: Remote Attach" configuration you have created, we create a docker-compose.debug.yml file with some different parameters to permit debugging.

As an example, here's a docker-compose.debug.yml file for an app I just created:

version: '3.4'

services:
  flasktest:
    image: flasktest
    build:
      context: .
      dockerfile: ./Dockerfile
    command: ["sh", "-c", "pip install debugpy -t /tmp && python /tmp/debugpy --wait-for-client --listen 0.0.0.0:5678 -m flask run --no-debugger --no-reload --host 0.0.0.0 --port 5000"]
    ports:
      - 5000:5000
      - 5678:5678
    environment:
      - FLASK_APP=manage.py

That command in particular is significantly different from what appears in docker-compose.yml; it installs debugpy (the debugger for Python), and launches it with the --wait-for-client flag so that the app won't start until you attach (which allows you to debug startup code).

For our normal Flask scenario, to debug you'd right click and Compose Up that docker-compose.debug.yml file, then F5 the Python remote attach config. Since your solution involves many services, I'd suggest putting that into a docker-compose.override.yml file instead, and using Compose Up from the command palette--which will automatically layer docker-compose.yml and then docker-compose.override.yml.

[ccrvlh]

@bwateratmsft let me see if understood your point correctly.

./.vscode/launch.json

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Remote Attach",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "/app"
                }
            ]
        }
    ]
}

./docker-compose.yml

version: "3.7"

services:

  app:
    build: .
    restart: always
    volumes:
      - .:/app
    env_file:
      - .env
    ports:
      - 8080:8080
      - 5678:5678

  worker:
    build: .
    restart: always
    command: celery -A app.worker --app=app.worker.celery worker --loglevel=DEBUG
    volumes:
      - .:/app
    links:
      - redis
      - rabbit
    env_file:
      - .env
    depends_on:     
      - redis
      - rabbit
      - app

  beat:
    build: .
    restart: always
    command: celery -A app.worker --app=app.worker.celery beat --loglevel=DEBUG
    volumes:
      - .:/app
    links:
      - redis
      - rabbit
    env_file:
      - .env
    depends_on:     
      - redis
      - rabbit
      - app

  flower:  
    image: mher/flower
    env_file:
      - .env    
    ports:  
      - 8888:8888 

  redis:
    image: redis:6.0-alpine
    volumes:
      - redis:/data

  rabbit:
    image: rabbitmq:3-management
    ports:
      - 5672:5672
      - 15672:15672

volumes:
  redis: {}

./docker-compose.override.yml

version: '3.4'

services:
  app:
    build:
      context: .
      dockerfile: ./Dockerfile
    command: ["sh", "-c", "pip install debugpy -t /tmp && python /tmp/debugpy --wait-for-client --listen 0.0.0.0:5678 -m app.uwsgi "]
    ports:
      - 5678:5678

Then I just right-click on my original ./docker-compose.yml and Docker Compose Up. After that I should Cmd + Shift + P to Docker: Add Dockerfile to Workspace (when I do this nothing happens).

When I run Docker Compose Up, and right afterwards (after the project is running) I press F5, the debugger starts loading, but right after it stops, and nothing happens, no error messages.

I'm definitely missing something, but I'm really confused as to what, and how to properly setup.
Any ideas?

Thanks!

[bwateratmsft]

At this point you won't need to do Docker: Add Dockerfiles to Workspace... anymore. A couple changes to make--

1. In docker-compose.yml, in ports, you can remove - 5678:5678; this port is only needed for debugging so it should only appear in docker-compose.override.yml.
2. In docker-compose.override.yml, you can remove the build section entirely

As for the command I don't know for sure how your app is started, I'll assume -m app.uwsgi is what you want, as you have written. For comparison, what we scaffold for a Flask app would be this:
command: ["sh", "-c", "pip install debugpy -t /tmp && python /tmp/debugpy --wait-for-client --listen 0.0.0.0:5678 -m flask run --no-debugger --no-reload --host 0.0.0.0 --port 5000"], along with environment: FLASK_APP=manage.py so Flask knows what to start.

The other thing to note is that the behavior will be different if you right click docker-compose.yml and do Compose Up, vs. opening the command palette (Ctrl + Shift + P), and doing Docker: Compose Up. If you right click the file, only that file will be used in the compose command; but if you run from the palette, we have some additional code that recognizes that you have docker-compose.yml and docker-compose.override.yml present, so we run the compose command without any -f <file> arguments--which causes compose to take on its default behavior, which will layer docker-compose.override.yml on top of docker-compose.yml.

In the case of debugging, you want to have that docker-compose.override.yml file be layered on top, so see if launching from the command palette and then F5'ing has the result you want.

[ucheNkadiCode]

Hey @lowercase00, would you mind providing us minimum repro steps from the start of your project? Or a repo? It would help us to recreate this scenario so we can move forward with looking into fixes.

We're also wondering if Compound Launch Configs are what you're looking for to debug multiple containers you want to attach to. Let us know 👍🏽

[bwateratmsft]

Changing this over to a Discussion, we want to try using that new feature from GitHub!