ci: add docker build action #549

vnghia · 2024-01-02T06:34:16Z

Part of #544

This PR adds a docker pipeline for PR, and push to main. Since we are no longer supporting bare-metal installation, I think we should probably just remove cargo-dist entirely and only focusing on Docker. What do you think @IgnisDa ?

IgnisDa · 2024-01-02T06:37:45Z

Yep cargo dist can be removed completely.

vnghia · 2024-01-02T07:21:32Z

@IgnisDa I think our .dockerignore is not strict enough and is causing a lot of cache invalidation. Could you point me to all the necessary files that we need to build the docker image (for example, I don't think we need the .github folder inside the Docker)? After this PR, we will build a lot more docker image than before and long building time could be a problem.

vnghia · 2024-01-02T07:24:09Z

The build successed on my end (https://github.com/vnghia/ryot/pkgs/container/ryot/162832490?tag=pr-3), this is probably because I don't have permission to write to this repo yet and the PR isn't merged.

Pushing to main will update ryot:unstable (https://github.com/vnghia/ryot/pkgs/container/ryot/162837319?tag=unstable)

Cargo.toml

IgnisDa · 2024-01-02T09:03:08Z

.github/workflows/release.yml

  # Mark the Github Release™ as a non-draft now that everything has succeeded!
  publish-release:
    # Only run after all the other tasks, but it's ok if upload-artifacts was skipped
    needs:
-      [create-release, upload-artifacts, docker-release, upload-kodi-plugin]
+      [create-release, upload-artifacts, upload-kodi-plugin]


Should we do away with cargo-dist in this release?

sorry, what do you mean by "do away" ?

Oh, I got it, never heard it before. The reason why I haven't touched it is because I don't have enough information about what it is doing, espcially the upload-kodi-plugin. Also the deploy and the build docs should probably be kept so what do you think about these two jobs ?

Yep deploy updates the public instance and upload-kodi-plugin creates the kodi archive. Both of the jobs are required.

vnghia · 2024-01-02T09:42:07Z

This PR also addresses #544 (comment)

IgnisDa · 2024-01-02T23:18:42Z

Help me understand this PR.

When I push a new tag, the Release workflow should now be responsible for creating the release and uploading the kodi artifact to it. Simultaneously, the Docker workflow creates the correct and uploads the image to ghcr.

If this is the case the only jobs in Release workflow should only have create-release, upload-kodi-plugin, publish-release, deploy, deploy-docs jobs. Currently, some of these jobs use cargo-dist, which we have to remove completely.

vnghia · 2024-01-03T03:31:13Z

Do you want the fly deploy to use the latest release or the latest commit to main ?

IgnisDa · 2024-01-03T04:25:26Z

The latest commit, what we tag as unstable as of now.

vnghia · 2024-01-03T06:48:53Z

I am not sure how we should handle the ci logic without cargo-dist, particularly the release vs pre-release so I think I will let you do the rest so it can fit with your needs.

IgnisDa · 2024-01-03T16:09:47Z

@vnghia I'm not sure how, but v4.1.0 got published by this PR (#557).

Is there any way to prevent changes from this PR being published to this repo?

vnghia · 2024-01-03T18:05:14Z

@IgnisDa, as far as I can tell, github will internally create a branch everytime there is a new PR. So when I do a release commit for testing on my fork, it will also make that commit a release on your end. It seems bypassing whatever permission model about the CI as well and this is a security issue with github and I will try to report it as soons as possible.

vnghia · 2024-01-04T02:45:31Z

It seems that you are the one who pushing the release commit, not me and thus bypassing the permission checking. Maybe because you are editing directly my PR so github counts that commit as yours.

IgnisDa · 2024-01-04T03:09:09Z

Ah well, that makes sense. I assumed that only workflows committed to main run.

vnghia · 2024-01-04T04:23:38Z

@IgnisDa, and my last suggestion: switch from supervisord to concurrently which will show logs as well as kill other processes if one dies. See vnghia#5. You can test by using ghcr.io/vnghia/ryot:pr-5.

and concurrently log

[backend] 2024-01-04T04:18:13.134363Z  INFO ryot: Running version: 4.0.8-beta.2
[proxy] {"level":"info","ts":1704341893.1851282,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
[proxy] {"level":"info","ts":1704341893.1903105,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
[proxy] {"level":"warn","ts":1704341893.1918602,"logger":"http.auto_https","msg":"automatic HTTPS is completely disabled for server","server_name":"srv0"}
[backend] 2024-01-04T04:18:13.192501Z  INFO sea_orm_migration::migrator: Applying all pending migrations
[proxy] {"level":"info","ts":1704341893.1934648,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003a9b00"}
[proxy] {"level":"info","ts":1704341893.1941092,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/home/ryot/.local/share/caddy"}
[proxy] {"level":"info","ts":1704341893.1944227,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
[proxy] {"level":"info","ts":1704341893.1955075,"msg":"autosaved config (load with --resume flag)","file":"/home/ryot/.config/caddy/autosave.json"}
[proxy] {"level":"info","ts":1704341893.195525,"msg":"serving initial configuration"}
[backend] 2024-01-04T04:18:13.196112Z ERROR ryot: Database migration failed: Custom Error: Migration file of version 'm20230901_create_metadata_group' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231025_add_collection_id_field_to_review' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231025_remove_duplicate_calendar_events' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231106_change_primary_key_exercise' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231113_add_media_ownership' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20230507_create_collection' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231101_change_production_status' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231105_change_calendar_event_column_type' is missing, this migration has been applied but its file is missing
[backend] Migration file of version 'm20231024_add_metadata_group_id_field_to_review' is missing, this migration has been applied but its file is missing
[proxy] {"level":"info","ts":1704341893.200983,"logger":"tls","msg":"finished cleaning storage units"}
[backend] Error: If this is a major version upgrade, please follow the instructions in the migration docs.
[backend] ryot exited with code 1
--> Sending SIGTERM to other processes..
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: spawn ps ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on ChildProcess instance at:
    at ChildProcess._handle.onexit (node:internal/child_process:292:12)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn ps',
  path: 'ps',
  spawnargs: [ '-o', 'pid', '--no-headers', '--ppid', 17 ]
}

vnghia · 2024-01-21T01:16:14Z

I've just made a tag here: https://github.com/vnghia/ryot/releases/tag/v4.2.0
I think the CI is falling because it is my own PR which don't have enough permission to push to your ghcr.io. After it is merged, that error should go away.

IgnisDa · 2024-01-21T15:24:10Z

@vnghia Do you think this is ready to be merged?

vnghia · 2024-01-21T18:54:31Z

I think it's ready to be merged 🥳

IgnisDa · 2024-01-22T00:51:53Z

@vnghia I am not able to figure out how to publish the latest tag. With the current configuration of the release workflow, that tag will not be published at all.

Can't have that happening since that will prevent all existing installations of Ryot (that use :latest tag) to not update at all.

Robin-Sch · 2024-01-27T22:22:33Z

@vnghia I am not able to figure out how to publish the latest tag. With the current configuration of the release workflow, that tag will not be published at all.

Can't have that happening since that will prevent all existing installations of Ryot (that use :latest tag) to not update at all.

Something like this should work for the publish to ghcr step:

tags: |
            ghcr.io/IgnisDa/ryot:latest
            ghcr.io/IgnisDa/ryot:${{ steps.meta.outputs.tags }}

I'm not sure if you can omit the ghcr.io/user/repo, but this is how I have it on my private repo.

IgnisDa · 2024-01-28T00:32:47Z

@vnghia I am not able to figure out how to publish the latest tag. With the current configuration of the release workflow, that tag will not be published at all.
Can't have that happening since that will prevent all existing installations of Ryot (that use :latest tag) to not update at all.

Something like this should work for the publish to ghcr step:
tags: |
            ghcr.io/IgnisDa/ryot:latest
            ghcr.io/IgnisDa/ryot:${{ steps.meta.outputs.tags }}
I'm not sure if you can omit the ghcr.io/user/repo, but this is how I have it on my private repo.

@Robin-Sch I want this workflow to make a :latest tag only if it is on the default branch (main). I think if I make the changes you proposed, it will release the latest tag for PRs too.

Robin-Sch · 2024-01-28T08:30:13Z

If a PR is made do you want a new docker version to be released at all? And do you want it to update the public fly instance?

You could make 2 seperate workflows, one for whatever you want on a PR, and one for whatever you want on a push.

IgnisDa · 2024-01-28T08:43:16Z

If a PR is made do you want a new docker version to be released at all? And do you want it to update the public fly instance?

You could make 2 seperate workflows, one for whatever you want on a PR, and one for whatever you want on a push.

On PRs I want a release called ghcr.io/ignisda/ryot:pr-505. When a commit is pushed to main, I want it to be released as ghcr.io/ignisda/ryot:unstable and the Fly instance be updated (it runs the unstable image). When a tag is pushed, I want it to be tagged as ghcr.io/ignisda/ryot:unstable and ghcr.io/ignisda/ryot:latest, and update the Fly instance.

Would these requirements mean I have to create 2 different workflows? Won't that result in a lot of duplicated code?

Robin-Sch · 2024-01-28T08:56:50Z

I'm not sure how you'd do it otherwise, maybe @vnghia could help with that?

EDIT:
You might be able to tweak https://github.com/vnghia/ryot/blob/c50989f96cc91c7e3f121c8359a8afe2c39b8059/.github/workflows/release.yml#L46

and add something like:

type=raw,value=latest,enable={{XXXXX}}

But I'm not sure what that XXXXX variable would be.

Also looking at the same workflow step, it seems like the current setup will also update vX.X.X whenever a new commit is made (at the same time that unstable is updated), as they use the same enable variable. Which I don't think is what you want?

IgnisDa · 2024-01-28T09:10:57Z

True, I do not want it.

@vnghia Could you please help here?

vnghia · 2024-01-28T10:44:51Z

I am a little busy right now. I will try on my fork next week!

Robin-Sch · 2024-02-16T15:53:29Z

Did you manage to find a solution?

IgnisDa · 2024-02-28T01:38:10Z

Hey @vnghia, any updates on this?

vnghia · 2024-02-28T02:00:04Z

I will be back next week. Sorry for the delay 😅

IgnisDa · 2024-04-18T23:49:27Z

Closing in favour of #770.

ci: add docker build action

7d31a75

vnghia added 2 commits January 2, 2024 13:52

docker: ignore .git for better cache

e17c2a2

ci: add missing permissions

87a0e85

vnghia added 3 commits January 2, 2024 14:35

cargo: remove cargo dist

12637b4

ci: cache to github action

bd813fc

ci: move docker-release to docker.yml

969051b

vnghia marked this pull request as ready for review January 2, 2024 08:20

vnghia force-pushed the docker-build-pr branch from 277a2ac to 9235d4b Compare January 2, 2024 08:47

ci: update image release tags

db4b54a

vnghia force-pushed the docker-build-pr branch from 9235d4b to db4b54a Compare January 2, 2024 08:52

IgnisDa requested changes Jan 2, 2024

View reviewed changes

cargo: merge profile dist to profile release

24b2f82

vnghia requested a review from IgnisDa January 2, 2024 09:44

Merge branch 'main' into docker-build-pr

e7f7f76

ci: move deploy to docker

1fe5be9

Merge branch 'main'

eb4c31a

ci: only run deploy if docker success

c466d39

IgnisDa added 2 commits January 15, 2024 18:32

ci: change image name

fac2f92

ci: deploy docs only on release

dd85cf8

IgnisDa added 4 commits January 21, 2024 20:56

Merge branch 'main'

840aea6

ci: remove changelog

a3ddedd

docs: remove section about changelogs

e12d13f

ci: remove useless files

b176cb0

IgnisDa added 2 commits January 22, 2024 06:07

Merge branch 'main'

ecddef9

ci: change order of services

7bee441

IgnisDa force-pushed the docker-build-pr branch from b1fafd4 to 7bee441 Compare January 22, 2024 00:39

IgnisDa added 3 commits January 22, 2024 06:12

docs: improve wording

e6cb3ec

docs: remove outdated section from dashboard

64d5b41

ci: do not release pr tag if on default branch

fec7880

Merge branch 'main'

c50989f

Merge branch 'main' into docker-build-pr

f103af0

IgnisDa closed this Apr 18, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: add docker build action #549

ci: add docker build action #549

vnghia commented Jan 2, 2024

IgnisDa commented Jan 2, 2024

vnghia commented Jan 2, 2024

vnghia commented Jan 2, 2024 •

edited

Loading

IgnisDa Jan 2, 2024

vnghia Jan 2, 2024

vnghia Jan 2, 2024 •

edited

Loading

IgnisDa Jan 2, 2024

vnghia commented Jan 2, 2024

IgnisDa commented Jan 2, 2024 •

edited

Loading

vnghia commented Jan 3, 2024

IgnisDa commented Jan 3, 2024

vnghia commented Jan 3, 2024

IgnisDa commented Jan 3, 2024

vnghia commented Jan 3, 2024

vnghia commented Jan 4, 2024

IgnisDa commented Jan 4, 2024

vnghia commented Jan 4, 2024 •

edited

Loading

vnghia commented Jan 21, 2024

IgnisDa commented Jan 21, 2024

vnghia commented Jan 21, 2024

IgnisDa commented Jan 22, 2024 •

edited

Loading

Robin-Sch commented Jan 27, 2024

IgnisDa commented Jan 28, 2024

Robin-Sch commented Jan 28, 2024

IgnisDa commented Jan 28, 2024

Robin-Sch commented Jan 28, 2024 •

edited

Loading

IgnisDa commented Jan 28, 2024

vnghia commented Jan 28, 2024

Robin-Sch commented Feb 16, 2024

IgnisDa commented Feb 28, 2024

vnghia commented Feb 28, 2024

IgnisDa commented Apr 18, 2024 •

edited

Loading

ci: add docker build action #549

ci: add docker build action #549

Conversation

vnghia commented Jan 2, 2024

IgnisDa commented Jan 2, 2024

vnghia commented Jan 2, 2024

vnghia commented Jan 2, 2024 • edited Loading

IgnisDa Jan 2, 2024

Choose a reason for hiding this comment

vnghia Jan 2, 2024

Choose a reason for hiding this comment

vnghia Jan 2, 2024 • edited Loading

Choose a reason for hiding this comment

IgnisDa Jan 2, 2024

Choose a reason for hiding this comment

vnghia commented Jan 2, 2024

IgnisDa commented Jan 2, 2024 • edited Loading

vnghia commented Jan 3, 2024

IgnisDa commented Jan 3, 2024

vnghia commented Jan 3, 2024

IgnisDa commented Jan 3, 2024

vnghia commented Jan 3, 2024

vnghia commented Jan 4, 2024

IgnisDa commented Jan 4, 2024

vnghia commented Jan 4, 2024 • edited Loading

vnghia commented Jan 21, 2024

IgnisDa commented Jan 21, 2024

vnghia commented Jan 21, 2024

IgnisDa commented Jan 22, 2024 • edited Loading

Robin-Sch commented Jan 27, 2024

IgnisDa commented Jan 28, 2024

Robin-Sch commented Jan 28, 2024

IgnisDa commented Jan 28, 2024

Robin-Sch commented Jan 28, 2024 • edited Loading

IgnisDa commented Jan 28, 2024

vnghia commented Jan 28, 2024

Robin-Sch commented Feb 16, 2024

IgnisDa commented Feb 28, 2024

vnghia commented Feb 28, 2024

IgnisDa commented Apr 18, 2024 • edited Loading

vnghia commented Jan 2, 2024 •

edited

Loading

vnghia Jan 2, 2024 •

edited

Loading

IgnisDa commented Jan 2, 2024 •

edited

Loading

vnghia commented Jan 4, 2024 •

edited

Loading

IgnisDa commented Jan 22, 2024 •

edited

Loading

Robin-Sch commented Jan 28, 2024 •

edited

Loading

IgnisDa commented Apr 18, 2024 •

edited

Loading