Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding Mac to S3 non-GPL FFmpeg compile #210

Merged
merged 1 commit into from
Sep 18, 2024
Merged

Conversation

scotts
Copy link
Contributor

@scotts scotts commented Sep 11, 2024

Adds CI workflows so that we build from S3 on Mac arm64 (the architecture for M1 processors). Enabling Mac arm64 builds required changes in several places:

  1. Modified fetch_and_expose_non_gpl_ffmpeg_libs.cmake to handle Mac OS on arm64. I had to generalize the logic so it can handle both Mac and Linux. This includes pulling the right tar files from S3 and specifying the right library names. (Linux uses .so; Mac uses .dylib.)
  2. Added .github/workflows/macos_wheel.yaml. The goal for this PR is to get it to the state where it builds from the S3 binaries and runs the Python tests. Subsequent PRs will turn this into the actual wheel building and testing.
  3. Renamed .github/workflows/wheel.yaml to linux_wheel.yaml.
  4. Modified test/utils.py so that when testing on Macs, we allow an absolute tolerance of 3 when comparing tensors. This means that we are okay with not having bit-for-bit-equality between Mac and Linux. Based on what I can find, I think trying to maintain bit-for-bit equality between systems and architectures is unrealistic. We will maintain bit-for-bit equality on Linux x86.
  5. Modified VideoDecoderOps.cpp to throw a pybind11 exception into one specific to PyTorch. The pybind11 exception was supposed to become a StopIteration exception in Python. That worked on Linux, but it segfaulted on Mac. PyTorch does not have an equivalent of StopIteration, but it does have one that becomes IndexError. I think that's close enough. From what I can tell, PyTorch does not always play nice with pybind11 exceptions; see END_HANDLE_TH_ERRORS_PYBIND prevents pybind11 Exception translation pytorch#34172. I opened Issue C++ code should not throw its own exceptions #212 to track using PyTorch errors everywhere in the C++ code.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Sep 11, 2024
@scotts scotts force-pushed the mac_s3_compile branch 29 times, most recently from a067323 to 3e78da5 Compare September 12, 2024 19:39
@scotts scotts force-pushed the mac_s3_compile branch 21 times, most recently from 109a966 to 139f9aa Compare September 17, 2024 18:19
@scotts scotts marked this pull request as ready for review September 18, 2024 00:42
test/utils.py Show resolved Hide resolved
test/utils.py Outdated Show resolved Hide resolved
@scotts scotts merged commit 707ea13 into pytorch:main Sep 18, 2024
20 of 21 checks passed
@scotts scotts deleted the mac_s3_compile branch September 18, 2024 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants