Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort sources in .sbf and when using globs for more reproducible outputs #2684

Merged
merged 2 commits into from
Jan 28, 2025

Conversation

echoix
Copy link
Contributor

@echoix echoix commented Jan 28, 2025

While comparing sdists and wheels created with a pyproject.toml or other build improvements, I noticed that some files had changes from commit to commit that could be avoided (in the .sbf files).
This is my first commit.

But why stop here? While not as useful as the first commit, any glob.glob() output that creates an output in the build log (when building) could benefit from a deterministic order.
This will help diff-ing logs between invocations to see what is changing when refactoring. When used for creating an archive, it will help creating a reproducible file. For example, adding files to a tar or zip archive with the same order is better than any order: the binary file would be more reproducible. I know it’s not enough to achieve Debian-level reproducible builds, like https://reproducible-builds.org/, since there are other things like timestamps and more not handled, but is a simple step towards this.

Note that I didn’t have time to compare the diff of the sdists and wheels with this since I committed yesterday night and let it run plus pulled and run the newest merged pull requests. (I also cherry picked my first commit for another branch and let it run too). It could be merged anyways, or maybe I will have the occasion to verify in the next two days.

This PR could be squash-merged, or normal merged as it only two commits, but for the same thing and the commits are maybe clean enough.

This will help diff-ing logs between invocations to see what is changing when refactoring. When used for creating an archive, it will help creating a reproducible file.
@echoix echoix changed the title Sort sources in .sbf and globs for more reproducible outputs Sort sources in .sbf and when using globs for more reproducible outputs Jan 28, 2025
@echoix
Copy link
Contributor Author

echoix commented Jan 28, 2025

So, apart from the files changed in this PR, plus the couple places that always change (version with a git suffix), the only changed files were the sip .sbf files. These were already changing between each build, but now

Artifacts of main at 1b5cc2e
(https://github.com/wxWidgets/Phoenix/actions/runs/13001425859) (left) compared to artifacts of this PR (https://github.com/wxWidgets/Phoenix/actions/runs/13017131171?pr=2684) (right), for sdist:
image
image

Now for the differences between the sdist of this PR (left), and the sdist of the PR of the same branch against my fork last night (right):
image
So, no changes other than the expected ones, and no difference in .sbf files.

I also spot checked some wheels, like the Windows 3.13 x64. Nothing useful changed, and all the binaries are different. By disassembling the native code (winmerge plugin using Microsoft (R) COFF/PE Dumper), there's no difference, but plain binaries you see that the random pip path used changes between files, and that is the most frequent difference.
image
image
The rest is only the checksums that differ, and the git version
image

Same thing for macOS 3.13 x64 (artifact name) wheel, the binaries differ where the temporary build path appears mostly, plus other that I can't decompile on Windows.

So its good to go!

@swt2c
Copy link
Collaborator

swt2c commented Jan 28, 2025

Nice work, thanks!

@swt2c swt2c merged commit 75c27ac into wxWidgets:master Jan 28, 2025
13 checks passed
@echoix
Copy link
Contributor Author

echoix commented Jan 28, 2025

I'll reupdate my minimal work for pyproject.toml, as I was close, except for the files contained in the sdist, there were more than before. It's my 4 different approach for it, and to make it safer and easier to compare with while migrating, I'm doing a first step that only targets having the existing builds work as before, not supporting pyproject.toml builds with other tools, as even 80 commits further it wasn't working enough and the risk was too big... So having a point where it is possible to revert to is important, and allows to keep the users build scripts the same.

@echoix
Copy link
Contributor Author

echoix commented Jan 31, 2025

@swt2c Do you know what's the best place to have a case to test wx.svg? I may have to change a bit how the extension module was made, and to have it at the same place I may have to rename and remove a duplicated prefix. I want to make sure it is importable the same way as before and test it. Otherwise I'm close to a PR

@swt2c
Copy link
Collaborator

swt2c commented Jan 31, 2025

It doesn't look like we have any existing unittests for wx.svg, but you could add one if you'd like.

There are some demos however that use it, such as demo/SVGImage_Bitmap.py and demo/SVGImage_Render.py.

@echoix
Copy link
Contributor Author

echoix commented Jan 31, 2025

I think a demo would do it ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants