Devcontainer support including Duckdb bootstrapped with minimal synthetic data #57

vvcb · 2024-09-20T17:20:42Z

As discussed at the dbt sync community call on 2024-09-20, this PR adds devcontainer support to the project.

Key features:

Python 3.12 with minimal requirements pre-installed for dbt to work with DuckDb.
Key VS Code extensions to support development including dbt-power-user extension (user may need to set preferred Python environment to ./dbt-env/bin/python3 to allow the extension to work - this requires a longer-term fix where the extension recognises this automatically.
dbt dependencies and pre-commit are installed at container creation time
DuckDb database created in ./data/ with minimal synthetic data loaded from seed files.

The user will have to run dbt run to create the rest of the models.

lawrenceadams

Nice work!! Feel free to ignore suggestions - but I think we can simplify some of the settings/extensions used here. Happy to debate

lawrenceadams · 2024-09-22T15:39:02Z

.devcontainer/devcontainer.json

+    // "forwardPorts": [],
+    // Use 'postCreateCommand' to run commands after the container is created.
+    "postCreateCommand": "bash ./.devcontainer/scripts/postCreate.sh",
+    "postStartCommand": "bash ./.devcontainer/scripts/postStart.sh",


This postStart script is not needed

Suggested change

"postStartCommand": "bash ./.devcontainer/scripts/postStart.sh",

Possibly - and this may be a relic from a fix to get devcontainers working for several projects on my machine (Windows with Docker Desktop using WSL2). If someone can confirm that git does not throw permission errors without that line, please feel free to remove it. It is borrowed from another devcontainer that does a bunch of stuff after starting.

The postStart.sh is needed in order to install the duckdb pieces.
I'm working off that shell script so that these happen after the Setup dbt section

I haven't added fancy logic to detect which OS.
My assumption is that for Github Codespaces that it is good enough for demoing for the symposium.
A longer term solution will be needed.

`

Setup dbt

echo "Setting up duckdb synthetic data directory"
mkdir ./data

echo "Installing Duckdb in Github Codespaces which is Ubuntu based OS"
wget https://github.com/duckdb/duckdb/releases/download/v1.0.0/duckdb_cli-linux-amd64.zip
unzip duckdb_cli-linux-amd64.zip

echo "Initialize the duckdb file and exit duckdb"
/workspaces/dbt-synthea/duckdb ./data/synthea_omop_etl.duckdb -s .quit

echo "Debugging dbt to check the connection to duckdb"
dbt debug
echo "Configure dbt dependent Python packages"
dbt deps
echo "Seed dbt duckdb with data"
dbt seed
echo "Compile and Build the dbt project"
dbt compile
dbt build
`

Update - My comment above is obviated by Lawrences awesome work to reconcile the python modules so that dbt debug checks both the connector to duckdb and initializes the file specified in the project yaml file.

lawrenceadams · 2024-09-22T15:43:04Z

.devcontainer/devcontainer.json

+                "dbt.dbtPythonPathOverride": "./dbt-env/bin/python3"
+            }
+        }
+    },


Valid json

Suggested change

},

}

lawrenceadams · 2024-09-22T15:44:58Z

.devcontainer/scripts/minimal_requirements.txt

+dbt-core==1.8.2
+dbt-duckdb==1.8.1
+pre-commit==3.7.1
+black
+python-dotenv


I think we don't need a requirements.txt file for this (if we only need to install dbt-duckdb)

Suggested change

dbt-core==1.8.2

dbt-duckdb==1.8.1

pre-commit==3.7.1

black

python-dotenv

I am unsure of the purpose of the root requirements.txt too? but I think thats for a different PR!

A devcontainer is for a consistent development experience - and black and pre-commit are the bare minimum and ensure that code contributions meet a minimum standard. We should have sqlfluff included explicitly in there too - I think it gets installed anyway as part of dbt.

I can see that the pre-commit -config.yaml file is very minimal - and can be improved as well. Possibly a subset of what is here -> https://github.com/vvcb/dbt-synthea/blob/vc/databricks/.pre-commit-config.yaml

Typically, one wouldn't need a separate requirements file within the devcontainer and the project requirements.txt should be sufficient. The new PR could address this with a single requirements.txt file for the project with database-specific dependencies and dependencies required only for development (such as black, pre-commit, etc.) specified as optional dependencies. This should allow the user to install the project with pip install dbt_synthea[postgres,dev] for instance.

Development dependencies are going to be different from dependencies required for just running the code.

Thank you for your explanation.

Ah that makes more sense, I trimmed out some of it as it wasn't referenced anywhere else and didn't see why it would be needed ~ it's probably fine to leave it in for now and can have another PR to tighten up precommit actions

can/should this be updated now that #67 is merged in?

lawrenceadams · 2024-09-22T15:46:05Z

.devcontainer/scripts/postCreate.sh

+# Set up git
+git config --global --add safe.directory /workspaces/dbt-synthea
+git config --global init.defaultBranch main
+


Happy to be corrected, but I think some of this is unnecessary - but maybe is required to defend against WSL2 instances (as I know git can get funny with Windows' ACLs)? Can leave in but I think not needed!

Regardless, the default branch doesn't need to be changed as the project already exists

Suggested change

# Set up git

git config --global --add safe.directory /workspaces/dbt-synthea

git config --global init.defaultBranch main

# Set up git

git config --global --add safe.directory /workspaces/dbt-synthea

git config --global init.defaultBranch main

Agree regarding default branch. But, the add safe.directory directive is required on when running this on WSL2.

.devcontainer/scripts/postCreate.sh

lawrenceadams · 2024-09-22T15:58:51Z

.devcontainer/scripts/postStart.sh

+git config --global --add safe.directory /workspaces/dbt-synthea
+


I don't think this needs to be run on every postStart (if it has already been done onCreate), if at all

Suggested change

git config --global --add safe.directory /workspaces/dbt-synthea

lawrenceadams · 2024-09-22T15:59:38Z

.vscode/settings.json

    "python.terminal.activateEnvInCurrentTerminal": true,
    "python.defaultInterpreterPath": "./dbt-env/bin/python3",
+    "dbt.dbtPythonPathOverride": "./dbt-env/bin/python3",


Suggested change

"python.terminal.activateEnvInCurrentTerminal": true,

"python.defaultInterpreterPath": "./dbt-env/bin/python3",

"dbt.dbtPythonPathOverride": "./dbt-env/bin/python3",

Hah 😄 I should have scrolled here before responding to the previous comment. Agree with this change - the user should be responsible for managing the environment rather than making this part of the codebase. But, this needs to be discussed with the others as it will change the behaviour of vscode for users.

so, i had set it up this way so that folks using VS Code would get their virtualenv activated each time they open the repo, without having to remember to activate it. if there's a way to instruct people to set this up without committing it to the repo, i'm open to doing that instead - is there?

Setting up and activating an environment should be the user's responsibility and outside of source control. What if someone wants to use uv or conda to manage their environments or use something other than vscode for development?

I would propose moving this to the documentation in README and modifying my devcontainer hack to get rid of setting up a virtual environment within a devcontainer - which as @lawrenceadams rightly pointed out, is redundant.

In this case should we add .vscode to .gitignore? (and then maybe we could add this settings json file to an extras folder so users can easily copy it to their local if they want to use it without thinking too much about it)

My preference is to keep .vscode under source control to share settings (eg. spaces Vs tabs, task configuration, etc.) - there is a good discussion here - https://stackoverflow.com/questions/32964920/should-i-commit-the-vscode-folder-to-source-control

However, environment activation steps can simply go in README rather than in code. I would expect the user to be able to setup a virtual environment if they are planning to use DuckDb, Postgres and dbt. If not, good time to learn anyway.

I will update this PR (with rebase).

Sounds good to me! Please go ahead and include that change in this PR :)

lawrenceadams · 2024-09-22T16:05:13Z

package-lock.yml

Forgot to bump this! Nice catch

lawrenceadams · 2024-09-22T16:10:20Z

.devcontainer/devcontainer.json

+            "extensions": [
+                "davidanson.vscode-markdownlint",
+                "editorconfig.editorconfig",
+                "mads-hartmann.bash-ide-vscode",
+                "mechatroner.rainbow-csv",
+                "ms-python.black-formatter",
+                "ms-python.python",
+                "ms-python.vscode-pylance",
+                "njpwerner.autodocstring",
+                "innoverio.vscode-dbt-power-user"
+            ],


Some of these are unused

Suggested change

"extensions": [

"davidanson.vscode-markdownlint",

"editorconfig.editorconfig",

"mads-hartmann.bash-ide-vscode",

"mechatroner.rainbow-csv",

"ms-python.black-formatter",

"ms-python.python",

"ms-python.vscode-pylance",

"njpwerner.autodocstring",

"innoverio.vscode-dbt-power-user"

],

"extensions": [

"davidanson.vscode-markdownlint",

"mechatroner.rainbow-csv",

"ms-python.python",

"ms-python.vscode-pylance",

"njpwerner.autodocstring",

"innoverio.vscode-dbt-power-user"

],

Are you sure you want to get rid of black?

I forgot there were 3 python files!

lawrenceadams · 2024-09-22T16:13:37Z

.devcontainer/devcontainer.json

+                "python.defaultInterpreterPath": "./dbt-env/bin/python3",
+                "dbt.dbtPythonPathOverride": "./dbt-env/bin/python3"


can trim down on unnecessary venv configs

Suggested change

"python.defaultInterpreterPath": "./dbt-env/bin/python3",

"dbt.dbtPythonPathOverride": "./dbt-env/bin/python3"

vvcb · 2024-09-22T17:01:46Z

@lawrenceadams , thank you for reviewing this. Agree with most things - but this was to get the devcontainer working with the existing codebase in preparation for the OHDSI symposium software demo to allow users to spin this up in their own GitHub codespaces.

Disagree with a few. Very happy for any of this to be modified as long as it works for everyone.

lawrenceadams · 2024-09-23T19:37:31Z

In all honesty is good to go for all intents and purposes, a lot of these issues are a reflection of python being annoying to manage packages with IMHO!

katy-sadowski · 2024-10-05T13:08:34Z

@vvcb thanks for working on this - i left a couple comments. also needs a rebase to incorporate recent changes 😄

i had been waiting for Heidi to take a look since Codespaces was her idea ... will ping in Teams to make sure she saw this.

heschmidt04

Please add the additions to postCreate.sh so that the duckdb install and the initialization of the database will allow the dbt commands to run properly.

heschmidt04 · 2024-10-11T17:26:24Z

.devcontainer/devcontainer.json

+    // "forwardPorts": [],
+    // Use 'postCreateCommand' to run commands after the container is created.
+    "postCreateCommand": "bash ./.devcontainer/scripts/postCreate.sh",
+    "postStartCommand": "bash ./.devcontainer/scripts/postStart.sh",


The postStart.sh is needed in order to install the duckdb pieces.
I'm working off that shell script so that these happen after the Setup dbt section

I haven't added fancy logic to detect which OS.
My assumption is that for Github Codespaces that it is good enough for demoing for the symposium.
A longer term solution will be needed.

`

Setup dbt

echo "Setting up duckdb synthetic data directory"
mkdir ./data

echo "Installing Duckdb in Github Codespaces which is Ubuntu based OS"
wget https://github.com/duckdb/duckdb/releases/download/v1.0.0/duckdb_cli-linux-amd64.zip
unzip duckdb_cli-linux-amd64.zip

echo "Initialize the duckdb file and exit duckdb"
/workspaces/dbt-synthea/duckdb ./data/synthea_omop_etl.duckdb -s .quit

echo "Debugging dbt to check the connection to duckdb"
dbt debug
echo "Configure dbt dependent Python packages"
dbt deps
echo "Seed dbt duckdb with data"
dbt seed
echo "Compile and Build the dbt project"
dbt compile
dbt build
`

vvcb added 2 commits September 20, 2024 15:12

Devcontainer setup

87122b4

Setup virtual environment within devcontainer

8288032

vvcb added the enhancement New feature or request label Sep 20, 2024

Fix dbt power user python path

72dc5c6

lawrenceadams self-requested a review September 22, 2024 15:38

lawrenceadams reviewed Sep 22, 2024

View reviewed changes

heschmidt04 requested changes Oct 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Devcontainer support including Duckdb bootstrapped with minimal synthetic data #57

Devcontainer support including Duckdb bootstrapped with minimal synthetic data #57

vvcb commented Sep 20, 2024

lawrenceadams left a comment

lawrenceadams Sep 22, 2024

vvcb Sep 22, 2024

heschmidt04 Oct 11, 2024

heschmidt04 Nov 7, 2024

lawrenceadams Sep 22, 2024

lawrenceadams Sep 22, 2024

lawrenceadams Sep 22, 2024

vvcb Sep 22, 2024

lawrenceadams Sep 23, 2024

katy-sadowski Oct 5, 2024

lawrenceadams Sep 22, 2024

vvcb Sep 22, 2024

lawrenceadams Sep 22, 2024

lawrenceadams Sep 22, 2024

vvcb Sep 22, 2024

katy-sadowski Oct 5, 2024

vvcb Oct 5, 2024

katy-sadowski Oct 5, 2024

vvcb Oct 12, 2024

katy-sadowski Oct 12, 2024

lawrenceadams Sep 22, 2024

lawrenceadams Sep 22, 2024

vvcb Sep 22, 2024

lawrenceadams Sep 23, 2024

lawrenceadams Sep 22, 2024

vvcb commented Sep 22, 2024

lawrenceadams commented Sep 23, 2024 •

edited

Loading

katy-sadowski commented Oct 5, 2024

heschmidt04 left a comment

heschmidt04 Oct 11, 2024

		git config --global --add safe.directory /workspaces/dbt-synthea

	"python.terminal.activateEnvInCurrentTerminal": true,
	"python.defaultInterpreterPath": "./dbt-env/bin/python3",
	"dbt.dbtPythonPathOverride": "./dbt-env/bin/python3",

Devcontainer support including Duckdb bootstrapped with minimal synthetic data #57

Are you sure you want to change the base?

Devcontainer support including Duckdb bootstrapped with minimal synthetic data #57

Conversation

vvcb commented Sep 20, 2024

lawrenceadams left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Setup dbt

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvcb commented Sep 22, 2024

lawrenceadams commented Sep 23, 2024 • edited Loading

katy-sadowski commented Oct 5, 2024

heschmidt04 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Setup dbt

lawrenceadams commented Sep 23, 2024 •

edited

Loading