Skip to content

Commit

Permalink
add pdm backend plugin (#2)
Browse files Browse the repository at this point in the history
  • Loading branch information
carderne authored Aug 20, 2024
1 parent ea0d14e commit 4a71b6f
Show file tree
Hide file tree
Showing 22 changed files with 325 additions and 46 deletions.
19 changes: 2 additions & 17 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ on:
release:
types: [published]
jobs:
publish-una:
publish-all:
environment: release
runs-on: ubuntu-latest
permissions:
Expand All @@ -14,20 +14,5 @@ jobs:
id: setup-rye
with:
version: '0.38.0'
- run: rye build
- uses: pypa/gh-action-pypi-publish@release/v1

publish-hatch-una:
environment: release
runs-on: ubuntu-latest
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
- uses: eifinger/setup-rye@v3
id: setup-rye
with:
version: '0.38.0'
- run: rye build
working-directory: plugins/hatch
- run: rye build --all
- uses: pypa/gh-action-pypi-publish@release/v1
25 changes: 17 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# una
# Una

<div align="center">
<img src="docs/assets/logo.svg" width="100">
Expand All @@ -20,25 +20,34 @@

</div>

una is a tool to make Python monorepos with Rye easier. It is a CLI tool and a Hatch plugin that does the following things:
Una is a tool to make Python monorepos with Rye easier. It is a CLI tool and a build plugin that does the following things:

1. Enable builds of individual apps or projects within a monorepo.
2. Ensure that internal and external dependencies are correctly specified.

una is inspired by [python-polylith](https://github.com/DavidVujic/python-polylith) and is based on that codebase.
But I find the [Polylith](https://polylith.gitbook.io/polylith) architecture to be quite intimidating for many, so wanted to create a lighter touch alternative that doesn't require too much re-thinking. This project has very limited ambitions and doesn't try to do everything a proper build system such as [Bazel](https://bazel.build/) or [Pants](https://www.pantsbuild.org/) does.
Una is inspired by [python-polylith](https://github.com/DavidVujic/python-polylith) and borrows extensively from that codebase.
But I find the [Polylith](https://polylith.gitbook.io/polylith) architecture to be quite intimidating for many, so wanted to create a lighter touch alternative that doesn't require too much re-thinking.
This project has very limited ambitions and doesn't try to do everything a proper build system such as [Bazel](https://bazel.build/) or [Pants](https://www.pantsbuild.org/) does.
It just tries to make a simple monorepo build just about possible.

una allows two directory structures or styles:
Una allows two directory structures or styles:
- `packages`: this is the default style, that is just some extra build help on top of a Rye workspace.
- `modules`: a more novel approach with just a single pyproject.toml, arguably better DevX and doesn't require a Rye workspace.

Within this context, we use the following words frequently:

- `lib`: a module or package that will be imported but not run.
- `app`: a module or package that will be run but never imported.
- `project`: a package with no code but only dependencies (only used in the `modules` style)

Currently it works with the following build backends, but more will follow:

- [Hatch](https://hatch.pypa.io) (used by default and and in all documentation)
- [PDM](https://pdm-project.org/)

## Examples
You can see examples for each of the two styles here:

- [una-example-packages](https://github.com/carderne/una-example-packages)
- [una-example-modules](https://github.com/carderne/una-example-modules)

Expand All @@ -52,7 +61,7 @@ cd unarepo
rye add --dev una
```

Then setup the una workspace. This will generate a structure and an example lib and app.
Then setup the Una workspace. This will generate a structure and an example lib and app.
```
rye run una create workspace
rye sync
Expand All @@ -66,7 +75,7 @@ tree
Have a look at the generated `__init__.py` files in the `apps/printer` and `libs/greeter` packages.
An external dependency ([cowsay-python](https://pypi.org/project/cowsay-python/)) has also been added to the latter's `pyproject.toml`.

The magic of `una` then comes in to resolve the graph of direct and transitive dependencies, which looks like this:
The magic of Una then comes in to resolve the graph of direct and transitive dependencies, which looks like this:
```elm
printer --> greeter --> cowsay-python
```
Expand Down Expand Up @@ -155,4 +164,4 @@ It covers additional things like:
- and more!

## License
una is distributed under the terms of the MIT license.
Una is distributed under the terms of the MIT license.
5 changes: 3 additions & 2 deletions docs/build.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Build

At build-time, `una` itself does nothing.
At build-time, Una itself does nothing.
This is when `hatch-una`, the plugin for [Hatch](https://hatch.pypa.io/) steps in and resolves the graph of dependencies.

Assuming your `pyproject.toml` is correctly configured, and your `[tool.una.libs]` section includes all necessary dependencies (_and_ transitive dependencies!),
Expand All @@ -17,4 +17,5 @@ rye build --wheel
rye run build
```

You'll get some `*.whl` files, which you can then deploy with Docker or whatever you prefer. They are fully self-contained, so you don't need Rye or `una` or anything else wherever you want to install them.
You'll get some `*.whl` files, which you can then deploy with Docker or whatever you prefer.
They are fully self-contained, so you don't need Rye or Una or anything else wherever you want to install them.
19 changes: 12 additions & 7 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# una
# Una

<div align="center">
<img src="assets/logo.svg" width="100">
Expand All @@ -20,18 +20,18 @@

</div>

una is a tool to make Python monorepos with Rye easier. It is a CLI tool and a Hatch plugin that does the following things:
Una is a tool to make Python monorepos with Rye easier. It is a CLI tool and a build plugin that does the following things:

1. Enable builds of individual apps or projects within a monorepo.
2. Ensure that internal and external dependencies are correctly specified.

una is inspired by [python-polylith](https://github.com/DavidVujic/python-polylith) and is based on that codebase.
But I find the [Polylith](https://polylith.gitbook.io/polylith) architecture to be quite intimidating for many, so wanted to create a lighter touch alternative that doesn't require too much re-thinking. This project has very limited ambitions and doesn't try to do everything a proper build system such as [Bazel](https://bazel.build/) or [Pants](https://www.pantsbuild.org/) does.
Una is inspired by [python-polylith](https://github.com/DavidVujic/python-polylith) and borrows extensively from that codebase.
But I find the [Polylith](https://polylith.gitbook.io/polylith) architecture to be quite intimidating for many, so wanted to create a lighter touch alternative that doesn't require too much re-thinking.
This project has very limited ambitions and doesn't try to do everything a proper build system such as [Bazel](https://bazel.build/) or [Pants](https://www.pantsbuild.org/) does.
It just tries to make a simple monorepo build just about possible.

una allows two directory structures or styles:

- `packages`: this is the lightest-touch approach, that is just some extra build help on top of a Rye workspace.
Una allows two directory structures or styles:
- `packages`: this is the default style, that is just some extra build help on top of a Rye workspace.
- `modules`: a more novel approach with just a single pyproject.toml, arguably better DevX and doesn't require a Rye workspace.

Within this context, we use the following words frequently:
Expand All @@ -40,6 +40,11 @@ Within this context, we use the following words frequently:
- `app`: a module or package that will be run but never imported.
- `project`: a package with no code but only dependencies (only used in the `modules` style)

Currently it works with the following build backends, but more will follow:

- [Hatch](https://hatch.pypa.io) (used by default and and in all documentation)
- [PDM](https://pdm-project.org/)

## Examples
You can see examples for each of the two styles here:

Expand Down
4 changes: 2 additions & 2 deletions docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ cd unarepo
rye add --dev una
```

Then setup the una workspace. This will generate a structure and an example lib and app.
Then setup the Una workspace. This will generate a structure and an example lib and app.
```
rye run una create workspace
rye sync
Expand All @@ -22,7 +22,7 @@ tree
Have a look at the generated `__init__.py` files in the `apps/printer` and `libs/greeter` packages.
An external dependency ([cowsay-python](https://pypi.org/project/cowsay-python/)) has also been added to the latter's `pyproject.toml`.

The magic of `una` then comes in to resolve the graph of direct and transitive dependencies, which looks like this:
The magic of Una then comes in to resolve the graph of direct and transitive dependencies, which looks like this:
```elm
printer --> greeter --> cowsay-python
```
Expand Down
4 changes: 2 additions & 2 deletions docs/style-modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,9 @@ The key differences are as follows:
So we add `projects/` directory. This contains no code, just a pyproject.toml and whatever else is needed to deploy the built project.
The pyproject will specify which internal dependencies are used in the project: exactly one app, and zero or more libs.
4. It must also specify all external dependencies that are used, including the transitive dependencies of internal libs that it uses.
But the una CLI will help with this!
But the Una CLI will help with this!

And there's one more benefit:
5. You can run pyright and pytest from the root directory!
This gives you a true monorepo benefit of having a single static analysis of the entire codebase.
But don't worry, una will help you to only test the bits that are needed.
But don't worry, Una will help you to only test the bits that are needed.
6 changes: 3 additions & 3 deletions docs/style-packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,11 @@ This means:
3. Type-checking and testing should be done on a per-package level.
That is, you should run `pyright` and `pytest` from `apps/server` or `libs/mylib`, _not_ from the root.

In the example above, the only build artifact will be for `apps/server`. At build-time, una will do the following:
In the example above, the only build artifact will be for `apps/server`. At build-time, Una will do the following:
1. Read the list of internal dependencies (more on this shortly) and inject them into the build.
2. Read all externel requirements of those dependencies, and add them to the dependency table.

You can then use the `una` CLI tool to ensure that all internal dependencies are kept in sync. What are the key steps?
You can then use the Una CLI tool to ensure that all internal dependencies are kept in sync. What are the key steps?
1. Use a Rye workspace:
```toml
# /pyproject.toml
Expand All @@ -64,7 +64,7 @@ build-backend = "hatchling.build"
[tool.una.libs]
"../../libs/mylib/example/mylib" = "example/mylib"
```
4. Then you can run `rye build --wheel` from that package directory and una will inject everything that is needed.
4. Then you can run `rye build --wheel` from that package directory and Una will inject everything that is needed.
Once you have your built `.whl` file, all you need in your Dockerfile is:
```Dockerfile
FROM python
Expand Down
4 changes: 3 additions & 1 deletion plugins/hatch/hatch_una/hatch_build.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@ def initialize(self, version: str, build_data: dict[str, Any]) -> None:
# need to add the root workspace pyproject.toml so that in src -> sdist -> wheel builds,
# we can still determine the style (for packages style)
add_root_pyproj = {
str(root_path / util.PYPROJ): str(util.EXTRA_PYPROJ / "root" / util.PYPROJ)
str(root_path / util.PYPROJ): str(
util.EXTRA_PYPROJ / util.ROOT_PYPROJ_SUBDIR / util.PYPROJ
)
}
if style == "packages":
add_packages_pyproj = {
Expand Down
Empty file.
5 changes: 3 additions & 2 deletions plugins/hatch/hatch_una/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
from typing import Any, Literal, TypeAlias

PYPROJ = "pyproject.toml"
EXTRA_PYPROJ = Path("extra_pyproj")
EXTRA_PYPROJ = Path("_extra_pyproj")
ROOT_PYPROJ_SUBDIR = Path("_root")


def load_conf(path: Path) -> dict[str, Any]:
Expand All @@ -24,7 +25,7 @@ def get_workspace_style(root_path: Path) -> Style:
# In builds that do src -> sdist -> wheel, the root pyproject.toml file will
# have been copied into the sdist so available for the wheel build.
# Here we check for both in order.
extra_root_path = EXTRA_PYPROJ / "root"
extra_root_path = EXTRA_PYPROJ / ROOT_PYPROJ_SUBDIR
if (root_path / PYPROJ).exists():
use_root_path = root_path
elif (extra_root_path / PYPROJ).exists():
Expand Down
2 changes: 1 addition & 1 deletion plugins/hatch/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ authors = [
readme = "README.md"
license = {text = "MIT License"}
requires-python = ">= 3.11"
keywords = ["rye", "monorepo", "build", "python"]
keywords = ["rye", "hatch", "monorepo", "build", "python"]

classifiers = [
"Environment :: Console",
Expand Down
5 changes: 5 additions & 0 deletions plugins/pdm/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# pdm-una

This is the PDM plugin for [una](https://github.com/carderne/una).

Read the full README there.
Empty file added plugins/pdm/pdm_una/__init__.py
Empty file.
9 changes: 9 additions & 0 deletions plugins/pdm/pdm_una/hook.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
from pdm.backend.hooks import Context

from pdm_una.include import force_include
from pdm_una.meta import add_dependencies


def pdm_build_initialize(context: Context):
add_dependencies(context)
force_include(context)
70 changes: 70 additions & 0 deletions plugins/pdm/pdm_una/include.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
from pathlib import Path

from pdm.backend.hooks import Context

from pdm_una import util


def force_include(context: Context) -> None:
"""
Force-include all needed internal monorepo dependencies.
"""
print("una: Injecting internal dependencies")
context.ensure_build_dir()

build_dir = context.build_dir

# load the config for this app/project
path = Path(context.root)
conf = util.load_conf(path)
name: str = conf["project"]["name"]

try:
int_deps: dict[str, str] = conf["tool"]["una"]["libs"]
except KeyError as e:
raise KeyError(
f"App/project '{name}' is missing '[tool.una.libs]' in pyproject.toml"
) from e

# need to determine workspace style (packages or modules)
# as packages style needs dependencies' pyproject.tomls to be included
# so that they're available in src -> sdist -> wheel builds
via_sdist = (util.EXTRA_PYPROJ / util.ROOT_PYPROJ_SUBDIR / util.PYPROJ).exists()
if via_sdist:
# nothing to do as everything should already be included in sdist...
return

root_path = path.parents[1]
style = util.get_workspace_style(root_path)

if not int_deps:
if style == "packages":
# this is fine, the app doesn't import anything internally
return
else:
# this is an empty project, useless and accidental
raise ValueError(f"Project '{name}' has no dependencies")

# make sure all int_deps exist
found = [Path(k) for k in int_deps if (path / k).exists()]
missing = set(int_deps) - set(str(p) for p in found)
if len(missing) > 0:
missing_str = ", ".join(missing)
raise ValueError(f"Could not find these paths: {missing_str}")

# need to add the root workspace pyproject.toml so that in src -> sdist -> wheel builds,
# we can still determine the style (for packages style)
util.copy_file(
root_path / util.PYPROJ,
build_dir / util.EXTRA_PYPROJ / util.ROOT_PYPROJ_SUBDIR / util.PYPROJ,
)
for src_str, dst_str in int_deps.items():
src = Path(src_str)
destination = build_dir / dst_str
util.copy_tree(src, destination)
if style == "packages":
# need these so src->sdist->wheel builds can access them for external deps
util.copy_file(
src.parents[1] / util.PYPROJ,
build_dir / util.EXTRA_PYPROJ / src.name / util.PYPROJ,
)
Loading

0 comments on commit 4a71b6f

Please sign in to comment.