Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement //example:docker #3

Merged
merged 14 commits into from
Mar 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .bazelrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
build --noexperimental_convenience_symlinks
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
/bazel-*
/bazel-*
/.vscode/
.pdm-python
10 changes: 5 additions & 5 deletions BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -3,16 +3,16 @@
# ----------------------------------------------------
# gazelle:prefix github.com/stackb/pycross_image

load("//tools/example_test:defs.bzl", "example_test_filegroup")
load("//bazel/tools/example_test:defs.bzl", "example_test_filegroup")

example_test_filegroup(
name = "example_test_files",
srcs = [
"BUILD.bazel",
"WORKSPACE",
"//bazel:example_test_files",
"//platform:example_test_files",
"//rules:example_test_files",
"//tools/example_test:example_test_files",
"//bazel/platforms:example_test_files",
"//bazel/rules:example_test_files",
"//bazel/tools/example_test:example_test_files",
"//bazel/workspace:example_test_files",
],
)
14 changes: 14 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.PHONY: pdm_sync
pdm_sync:
bazel run //bazel/tools/pdm -- sync
rm -rf .venv

.PHONY: pdm_lock
pdm_lock:
bazel run //bazel/tools/pdm -- lock --static-urls
rm -rf .venv

.PHONY: example_oci_pdm_add
example_oci_pdm_add:
PDM_PROJECT=$(PWD)/example/oci bazel run //bazel/tools/pdm -- add grpclib
rm -rf example/oci/.venv
96 changes: 91 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,95 @@
# pycross_image
# @pycross_image

`pycross_image` provides starlark rules for building container images for python
applications with bazel.
![build-status](https://github.com/stackb/pycross_image/actions/workflows/ci.yaml/badge.svg)

You can use the rules directly from this repo, or simply as examples and
copy/paste them into your own project.
Bazel starlark rules for building container images from `py_binary` :sparkles:
using [@rules_pycross](https://github.com/jvolkman/rules_pycross) :magic:.

`@pycross_image` provides:

- `load("@pycross_image//bazel/rules:oci.bzl", "py_image")`: image rule
compatible with [@rules_oci](https://github.com/bazel-contrib/rules_oci)
- `load("@pycross_image//bazel/rules:docker.bzl", "py_image")`: image rule
compatible with [@rules_docker](https://github.com/bazelbuild/rules_docker)

## Installation & Usage

See [releases] page for an `http_archive` of the latest `@pycross_image`.

Examples:
- [@rules_oci example](example/oci/WORKSPACE.in).
- [@rules_docker example](example/docker/WORKSPACE.in).

A few notes about the workspace setup:

- It's divided into "steps" based on load statement dependencies. `step1.bzl`
only depends on things declared in `repositories.bzl`, `step2.bzl` depends on
things declared in `step3.bzl`, etc (this pattern is from
[tensorflow](https://github.com/tensorflow/tensorflow/tree/master/tensorflow)).
- Your workspace may already have many of the dependencies. Some of the
external workspace names may be differ from yours. Use the example as a study
guide rather than canonical reference.
- The examples are only tested with the older `WORKSPACE`. The rules may not be
compatible with bzlmod yet.

## How it Works

```mermaid
graph TD;
pypi[(pypi)]
DefaultInfo[[DefaultInfo]]
numpy{{numpy}}
grpclib{{grpclib}}

subgraph linux_x86_64["linuxx86_64\n"]
image.tar-->image;
image-->app_layer;
image-->site_packages_layer;
image-->interpreter_layer;
app_layer-->DefaultInfo;
site_packages_layer-->DefaultInfo;
interpreter_layer-->DefaultInfo;
end

DefaultInfo--transition :linux_x86_64-->pycross_binary;
pycross_binary-->py_binary;
py_binary-->numpy;
py_binary-->grpclib;

numpy-.->numpy-cp310-macosx_arm64.whl
numpy-.->numpy-cp310-manylinux_x86_64.whl
grpclib-.->grpclib.whl
numpy-cp310-macosx_arm64.whl-->pypi
numpy-cp310-manylinux_x86_64.whl-->pypi

subgraph zig_toolchain
grpclib.whl-->grpclib-tar.gz
end

grpclib-tar.gz-->pypi

style pypi fill:#3171b2,stroke:#333
style numpy fill:#5d97d2,stroke:#3171b2,color:black
style grpclib fill:#5d97d2,stroke:#3171b2,color:black
style grpclib.whl stroke:#bc082b

style numpy-cp310-manylinux_x86_64.whl stroke:#bc082b
style numpy-cp310-macosx_arm64.whl stroke:#bb22d8
style linux_x86_64 stroke:#bc082b,fill:none
```

In this example the `py_binary` rule has `deps` on two python wheels:
- `numpy` has a binary wheel available from pypi for both the darwin and linux
platforms.
- `grpclib` only has a source distribution available.

The `pycross_binary` rule transitions from the host platform to `:linux_x86_64`.
- the transition affects how `@rules_pycross` fetches wheels. If the binary
distribution is available, take it.
- if the binary distribution is not available, compile from source using (in
this case, with `zig` and `@hermetic_cc_toolchains`).

The image is partitioned into three tar layers by matching against filename
patterns (see rule implementation for details).

> `@rules_pycross` supports dependency fetching using PDM or poetry.
61 changes: 22 additions & 39 deletions WORKSPACE
Original file line number Diff line number Diff line change
@@ -1,68 +1,51 @@
workspace(name = "pycross_image")

load("@pycross_image//bazel:repositories.bzl", "repositories")
# -----------------------------------------

load("@pycross_image//bazel/workspace:repositories.bzl", "repositories")

repositories()

# -----------------------------------------

load(
"@pycross_image//bazel:workspace0.bzl",
"setup_aspect_bazel_lib",
"setup_bazel_gazelle",
"setup_bazel_skylib",
"setup_rules_docker",
"setup_rules_go",
"setup_rules_oci",
"setup_rules_python",
"setup_zig_toolchains",
)
load("@pycross_image//bazel/workspace:step1.bzl", "step1")

step1.setup_zig_toolchains()

setup_zig_toolchains()
step1.setup_rules_go()

setup_rules_go()
step1.setup_bazel_gazelle()

setup_bazel_gazelle()
step1.setup_bazel_skylib()

setup_bazel_skylib()
step1.setup_rules_python()

setup_rules_python()
step1.setup_aspect_bazel_lib()

setup_aspect_bazel_lib()
step1.setup_rules_oci()

setup_rules_oci()
step1.setup_rules_docker()

setup_rules_docker()
step1.setup_container_structure_test()

# -----------------------------------------

load(
"@pycross_image//bazel:workspace1.bzl",
"setup_docker_containers",
"setup_oci_containers",
"setup_rules_pycross",
)
load("@pycross_image//bazel/workspace:step2.bzl", "step2")

setup_oci_containers()
step2.setup_pycross_image_base_oci()

setup_docker_containers()
step2.setup_pycross_image_base_container()

setup_rules_pycross()
step2.setup_rules_pycross()

# -----------------------------------------

load(
"@pycross_image//bazel:workspace2.bzl",
"setup_pdm_deps",
)
load("@pycross_image//bazel/workspace:step3.bzl", "step3")

setup_pdm_deps()
step3.setup_pypi_deps_for_pdm()

# -----------------------------------------

load(
"@pycross_image//bazel:workspace3.bzl",
"install_pdm_deps",
)
load("@pycross_image//bazel/workspace:step4.bzl", "step4")

install_pdm_deps()
step4.install_pypi_deps_for_pdm()
14 changes: 0 additions & 14 deletions bazel/BUILD.bazel

This file was deleted.

3 changes: 0 additions & 3 deletions bazel/container.bzl

This file was deleted.

33 changes: 33 additions & 0 deletions bazel/platforms/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
load("//bazel/tools/example_test:defs.bzl", "example_test_filegroup")

platform(
name = "linux_x86_64",
constraint_values = [
"@platforms//os:linux",
"@platforms//cpu:x86_64",
],
visibility = ["//visibility:public"],
)

platform(
name = "darwin_x86_64",
constraint_values = [
"@platforms//os:macos",
"@platforms//cpu:x86_64",
],
visibility = ["//visibility:public"],
)

platform(
name = "darwin_arm64",
constraint_values = [
"@platforms//os:macos",
"@platforms//cpu:arm64",
],
visibility = ["//visibility:public"],
)

example_test_filegroup(
name = "example_test_files",
srcs = ["BUILD.bazel"],
)
5 changes: 0 additions & 5 deletions bazel/python/deps/BUILD.bazel

This file was deleted.

15 changes: 0 additions & 15 deletions bazel/python/pdm/BUILD.bazel

This file was deleted.

36 changes: 0 additions & 36 deletions bazel/python/pycross_image.md

This file was deleted.

4 changes: 2 additions & 2 deletions rules/BUILD.bazel → bazel/rules/BUILD.bazel
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
load("//tools/example_test:defs.bzl", "example_test_filegroup")
load("//bazel/tools/example_test:defs.bzl", "example_test_filegroup")

example_test_filegroup(
name = "example_test_files",
srcs = [
"BUILD.bazel",
"docker.bzl",
"oci.bzl",
"py_layers.bzl",
"support.bzl",
],
)
Loading
Loading