Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into alext_initialize_ne…
Browse files Browse the repository at this point in the history
…w_array
  • Loading branch information
SonicField committed May 8, 2024
2 parents 451f0ee + c4f9823 commit 1d7c44f
Show file tree
Hide file tree
Showing 605 changed files with 26,230 additions and 6,605 deletions.
12 changes: 12 additions & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,18 @@ Lib/test/support/interpreters/ @ericsnowcurrently
Modules/_xx*interp*module.c @ericsnowcurrently
Lib/test/test_interpreters/ @ericsnowcurrently

# Android
**/*Android* @mhsmith
**/*android* @mhsmith

# iOS (but not termios)
**/iOS* @freakboy3742
**/ios* @freakboy3742
**/*_iOS* @freakboy3742
**/*_ios* @freakboy3742
**/*-iOS* @freakboy3742
**/*-ios* @freakboy3742

# WebAssembly
/Tools/wasm/ @brettcannon

Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,7 @@ jobs:
with:
config_hash: ${{ needs.check_source.outputs.config_hash }}
free-threading: true
# macos-14 is M1
# macos-14-large is Intel with 12 cores (most parallelism)
os-matrix: '["macos-14"]'

build_ubuntu:
Expand Down Expand Up @@ -250,7 +250,7 @@ jobs:
build_ubuntu_ssltests:
name: 'Ubuntu SSL tests with OpenSSL'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
timeout-minutes: 60
needs: check_source
if: needs.check_source.outputs.run_tests == 'true'
Expand Down Expand Up @@ -316,7 +316,7 @@ jobs:

test_hypothesis:
name: "Hypothesis tests on Ubuntu"
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
timeout-minutes: 60
needs: check_source
if: needs.check_source.outputs.run_tests == 'true' && needs.check_source.outputs.run_hypothesis == 'true'
Expand Down Expand Up @@ -429,7 +429,7 @@ jobs:

build_asan:
name: 'Address sanitizer'
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
timeout-minutes: 60
needs: check_source
if: needs.check_source.outputs.run_tests == 'true'
Expand Down
26 changes: 14 additions & 12 deletions .github/workflows/jit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
jit:
name: ${{ matrix.target }} (${{ matrix.debug && 'Debug' || 'Release' }})
runs-on: ${{ matrix.runner }}
timeout-minutes: 75
timeout-minutes: 90
strategy:
fail-fast: false
matrix:
Expand All @@ -41,7 +41,7 @@ jobs:
- true
- false
llvm:
- 16
- 18
include:
- target: i686-pc-windows-msvc/msvc
architecture: Win32
Expand Down Expand Up @@ -75,14 +75,10 @@ jobs:
architecture: aarch64
runner: ubuntu-latest
compiler: gcc
# These fail because of emulation, not because of the JIT:
exclude: test_pathlib test_posixpath test_unix_events test_init test_process_pool test_shutdown test_multiprocessing_fork test_cmd_line test_faulthandler test_os test_perf_profiler test_posix test_signal test_socket test_subprocess test_threading test_venv test_external_inspection
- target: aarch64-unknown-linux-gnu/clang
architecture: aarch64
runner: ubuntu-latest
compiler: clang
# These fail because of emulation, not because of the JIT:
exclude: test_pathlib test_posixpath test_unix_events test_init test_process_pool test_shutdown test_multiprocessing_fork test_cmd_line test_faulthandler test_os test_perf_profiler test_posix test_signal test_socket test_subprocess test_threading test_venv test_external_inspection
env:
CC: ${{ matrix.compiler }}
steps:
Expand All @@ -94,37 +90,43 @@ jobs:
- name: Native Windows
if: runner.os == 'Windows' && matrix.architecture != 'ARM64'
run: |
choco upgrade llvm -y
choco install llvm --allow-downgrade --no-progress --version ${{ matrix.llvm }}
./PCbuild/build.bat --experimental-jit ${{ matrix.debug && '-d' || '--pgo' }} -p ${{ matrix.architecture }}
./PCbuild/rt.bat ${{ matrix.debug && '-d' }} -p ${{ matrix.architecture }} -q --exclude ${{ matrix.exclude }} --multiprocess 0 --timeout 4500 --verbose2 --verbose3
./PCbuild/rt.bat ${{ matrix.debug && '-d' || '' }} -p ${{ matrix.architecture }} -q --multiprocess 0 --timeout 4500 --verbose2 --verbose3
# No PGO or tests (yet):
- name: Emulated Windows
if: runner.os == 'Windows' && matrix.architecture == 'ARM64'
run: |
choco upgrade llvm -y
choco install llvm --allow-downgrade --no-progress --version ${{ matrix.llvm }}
./PCbuild/build.bat --experimental-jit ${{ matrix.debug && '-d' || '' }} -p ${{ matrix.architecture }}
- name: Native macOS
if: runner.os == 'macOS'
run: |
brew update
brew install llvm@${{ matrix.llvm }}
SDKROOT="$(xcrun --show-sdk-path)" \
./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '--enable-optimizations --with-lto' }}
make all --jobs 4
./python.exe -m test --exclude ${{ matrix.exclude }} --multiprocess 0 --timeout 4500 --verbose2 --verbose3
./python.exe -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3
# --with-lto has been removed temporarily as a result of an open issue in LLVM 18 (see https://github.com/llvm/llvm-project/issues/87553)
- name: Native Linux
if: runner.os == 'Linux' && matrix.architecture == 'x86_64'
run: |
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" ./llvm.sh ${{ matrix.llvm }}
export PATH="$(llvm-config-${{ matrix.llvm }} --bindir):$PATH"
./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '--enable-optimizations --with-lto' }}
./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '--enable-optimizations' }}
make all --jobs 4
./python -m test --exclude ${{ matrix.exclude }} --multiprocess 0 --timeout 4500 --verbose2 --verbose3
./python -m test --multiprocess 0 --timeout 4500 --verbose2 --verbose3
# --with-lto has been removed temporarily as a result of an open issue in LLVM 18 (see https://github.com/llvm/llvm-project/issues/87553)
- name: Emulated Linux
if: runner.os == 'Linux' && matrix.architecture != 'x86_64'
# The --ignorefile on ./python -m test is used to exclude tests known to fail when running on an emulated Linux.
run: |
sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" ./llvm.sh ${{ matrix.llvm }}
export PATH="$(llvm-config-${{ matrix.llvm }} --bindir):$PATH"
Expand All @@ -139,6 +141,6 @@ jobs:
CC="${{ matrix.compiler == 'clang' && 'clang --target=$HOST' || '$HOST-gcc' }}" \
CPP="$CC --preprocess" \
HOSTRUNNER=qemu-${{ matrix.architecture }} \
./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '--enable-optimizations --with-lto' }} --build=x86_64-linux-gnu --host="$HOST" --with-build-python=../build/bin/python3 --with-pkg-config=no ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptc=no ac_cv_file__dev_ptmx=yes
./configure --enable-experimental-jit ${{ matrix.debug && '--with-pydebug' || '--enable-optimizations ' }} --build=x86_64-linux-gnu --host="$HOST" --with-build-python=../build/bin/python3 --with-pkg-config=no ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptc=no ac_cv_file__dev_ptmx=yes
make all --jobs 4
./python -m test --exclude ${{ matrix.exclude }} --multiprocess 0 --timeout 4500 --verbose2 --verbose3
./python -m test --ignorefile=Tools/jit/ignore-tests-emulated-linux.txt --multiprocess 0 --timeout 4500 --verbose2 --verbose3
4 changes: 3 additions & 1 deletion .github/workflows/mypy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ on:
pull_request:
paths:
- ".github/workflows/mypy.yml"
- "Lib/_pyrepl/**"
- "Lib/test/libregrtest/**"
- "Tools/build/generate_sbom.py"
- "Tools/cases_generator/**"
Expand Down Expand Up @@ -35,8 +36,9 @@ jobs:
strategy:
matrix:
target: [
"Lib/_pyrepl",
"Lib/test/libregrtest",
"Tools/build/",
"Tools/build",
"Tools/cases_generator",
"Tools/clinic",
"Tools/jit",
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/posix-deps-apt.sh
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ apt-get -yq install \
libgdbm-dev \
libgdbm-compat-dev \
liblzma-dev \
libmpdec-dev \
libncurses5-dev \
libreadline6-dev \
libsqlite3-dev \
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/reusable-macos.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ jobs:
HOMEBREW_NO_INSTALL_CLEANUP: 1
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1
PYTHONSTRICTEXTENSIONBUILD: 1
TERM: linux
strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -49,7 +50,7 @@ jobs:
--prefix=/opt/python-dev \
--with-openssl="$(brew --prefix [email protected])"
- name: Build CPython
run: make -j4
run: make -j8
- name: Display build info
run: make pythoninfo
- name: Tests
Expand Down
9 changes: 8 additions & 1 deletion .github/workflows/reusable-tsan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,14 @@ jobs:
- name: Install Dependencies
run: |
sudo ./.github/workflows/posix-deps-apt.sh
sudo apt install -y clang
# Install clang-18
wget https://apt.llvm.org/llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 18
sudo update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100
sudo update-alternatives --set clang /usr/bin/clang-18
sudo update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-18 100
sudo update-alternatives --set clang++ /usr/bin/clang++-18
# Reduce ASLR to avoid TSAN crashing
sudo sysctl -w vm.mmap_rnd_bits=28
- name: TSAN Option Setup
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/reusable-ubuntu.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,12 @@ jobs:
build_ubuntu_reusable:
name: 'build and test'
timeout-minutes: 60
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
env:
FORCE_COLOR: 1
OPENSSL_VER: 3.0.13
PYTHONSTRICTEXTENSIONBUILD: 1
TERM: linux
steps:
- uses: actions/checkout@v4
- name: Register gcc problem matcher
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/reusable-wasi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
build_wasi_reusable:
name: 'build and test'
timeout-minutes: 60
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
env:
WASMTIME_VERSION: 18.0.3
WASI_SDK_VERSION: 21
Expand Down
8 changes: 8 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,14 @@ repos:
args: [--exit-non-zero-on-fix, --config=Tools/clinic/.ruff.toml]
files: ^Tools/clinic/|Lib/test/test_clinic.py

- repo: https://github.com/psf/black-pre-commit-mirror
rev: 24.4.2
hooks:
- id: black
name: Run Black on Tools/jit/
files: ^Tools/jit/
language_version: python3.12

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
Expand Down
43 changes: 38 additions & 5 deletions Android/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,25 @@ you don't already have the SDK, here's how to install it:
`android-sdk/cmdline-tools/latest`.
* `export ANDROID_HOME=/path/to/android-sdk`

The `android.py` script also requires the following commands to be on the `PATH`:

* `curl`
* `java`
* `tar`
* `unzip`


## Building

Building for Android requires doing a cross-build where you have a "build"
Python to help produce an Android build of CPython. This procedure has been
tested on Linux and macOS.
Python can be built for Android on any POSIX platform supported by the Android
development tools, which currently means Linux or macOS. This involves doing a
cross-build where you use a "build" Python (for your development machine) to
help produce a "host" Python for Android.

First, make sure you have all the usual tools and libraries needed to build
Python for your development machine. The only Android tool you need to install
is the command line tools package above: the build script will download the
rest.

The easiest way to do a build is to use the `android.py` script. You can either
have it perform the entire build process from start to finish in one step, or
Expand All @@ -43,9 +56,10 @@ The discrete steps for building via `android.py` are:
./android.py make-host HOST
```

To see the possible values of HOST, run `./android.py configure-host --help`.
`HOST` identifies which architecture to build. To see the possible values, run
`./android.py configure-host --help`.

Or to do it all in a single command, run:
To do all steps in a single command, run:

```sh
./android.py build HOST
Expand All @@ -62,3 +76,22 @@ call. For example, if you want a pydebug build that also caches the results from
```sh
./android.py build HOST -- -C --with-pydebug
```


## Testing

To run the Python test suite on Android:

* Install Android Studio, if you don't already have it.
* Follow the instructions in the previous section to build all supported
architectures.
* Run `./android.py setup-testbed` to download the Gradle wrapper.
* Open the `testbed` directory in Android Studio.
* In the *Device Manager* dock, connect a device or start an emulator.
Then select it from the drop-down list in the toolbar.
* Click the "Run" button in the toolbar.
* The testbed app displays nothing on screen while running. To see its output,
open the [Logcat window](https://developer.android.com/studio/debug/logcat).

To run specific tests, or pass any other arguments to the test suite, edit the
command line in testbed/app/src/main/python/main.py.
6 changes: 6 additions & 0 deletions Android/android-env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ done
export CFLAGS=""
export LDFLAGS="-Wl,--build-id=sha1 -Wl,--no-rosegment"

# Unlike Linux, Android does not implicitly use a dlopened library to resolve
# relocations in subsequently-loaded libraries, even if RTLD_GLOBAL is used
# (https://github.com/android/ndk/issues/1244). So any library that fails to
# build with this flag, would also fail to load at runtime.
LDFLAGS="$LDFLAGS -Wl,--no-undefined"

# Many packages get away with omitting -lm on Linux, but Android is stricter.
LDFLAGS="$LDFLAGS -lm"

Expand Down
Loading

0 comments on commit 1d7c44f

Please sign in to comment.