Skip to content

Commit

Permalink
Merge branch 'master' into deploy
Browse files Browse the repository at this point in the history
  • Loading branch information
mpe committed Aug 12, 2024
2 parents 9c775c5 + f67b730 commit 5ee9aeb
Show file tree
Hide file tree
Showing 156 changed files with 1,958 additions and 773 deletions.
69 changes: 69 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,3 +149,72 @@ $ make rebuild-image@ppc64le@ubuntu

Note that the build mounts the source tree read-only, so nothing it does can
affect your source tree.

Bisecting the kernel vs a selftest
----------------------------------

Build the selftests using a version of the test that's known good. Usually
there's no reason to rebuild the tests on every kernel revision.

These examples are run from the kernel directory, not the ci-scripts directory.
It can be done either way, but it's more natural to run from the kernel
directory when bisectting the kernel. This assumes Linux is in `~/linux` and
these scripts are in `~/ci-scripts`, adapt as appropriate.

```
$ cd ~/linux
```

```
$ make SRC=$PWD -C ~/ci-scripts/build QUIET=1 JFACTOR=$(nproc) ppctests@ppc64le@[email protected] INSTALL=1
```

:rotating_light: Using an old Ubuntu image uses an older libc, which is more
likely to be present on the guest root filesystem. Another option is to build
the selftests statically.

Tar up the selftests into the current directory, the qemu scripts will detect them:

```
$ tar -czf selftests.tar.gz -C $HOME/ci-scripts/build/output/selftests_powerpc@ppc64le@[email protected]/ install
```

```
$ ~/ci-scripts/scripts/boot/qemu-pseries+kvm --callback "run_selftests(powerpc/mm:wild_bctr)"
...
INFO: Running 'qemu-system-ppc64 -nographic -vga none -M pseries -smp 8 -m 4G -accel kvm ...
...
/ # INFO: Running individual selftests powerpc/mm:wild_bctr
/var/tmp/selftests/run_kselftest.sh -t powerpc/mm:wild_bctr
[ 2.783761][ T201] kselftest: Running tests in powerpc
TAP version 13
1..1
# timeout set to 300
# selftests: powerpc/mm: wild_bctr
# test: wild_bctr
# tags: git_version:v6.8-rc6-2555-gfe559db
# Everything is OK in here.
...
# success: wild_bctr
ok 1 selftests: powerpc/mm: wild_bctr
/ # poweroff
/ # Stopping network: [ 3.104385][ T274] ip (274) used greatest stack depth: 10912 bytes left
OK
Saving random seed: OK
Stopping klogd: OK
Stopping syslogd: OK
umount: devtmpfs busy - remounted read-only
umount: can't unmount /: Invalid argument
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[ 5.152672][ T293] reboot: Power down
INFO: Test completed OK
```

More than one selftest can be run by passing multiple arguments to
`run_selftests` or by passing multiple `--callback` options.

From there the bisection can either be run by hand, or fully automated by
creating a script to build the kernel and run the qemu test.
8 changes: 2 additions & 6 deletions build/allcross/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ RUN apt-get -q -y update && \
libc-dev \
libelf-dev \
libfuse-dev \
liblz4-tool \
lz4 \
libssl-dev \
lzop \
make \
Expand All @@ -52,11 +52,7 @@ RUN apt-get -q -y update && \
xz-utils && \
rm -rf /var/lib/apt/lists/* /tmp/packages.sh /var/cache/* /var/log/dpkg.log

ARG uid
ARG gid

RUN groupadd --gid $gid linuxppc
RUN useradd --uid $uid --gid $gid linuxppc
RUN useradd linuxppc
USER linuxppc

COPY scripts/container-build.sh /bin/container-build.sh
Expand Down
6 changes: 1 addition & 5 deletions build/docs/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,7 @@ RUN apt-get -q -y update && \
&& \
rm -rf /var/lib/apt/lists/* /var/cache/* /var/log/dpkg.log

ARG uid
ARG gid

RUN groupadd --gid $gid linuxppc
RUN useradd --uid $uid --gid $gid linuxppc
RUN useradd linuxppc
USER linuxppc

COPY scripts/container-build.sh /bin/container-build.sh
Expand Down
6 changes: 1 addition & 5 deletions build/fedora/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,7 @@ RUN dnf -y install --setopt=install_weak_deps=False \
COPY fedora/make-links.sh /tmp/make-links.sh
RUN /tmp/make-links.sh ${compiler_version} && rm /tmp/make-links.sh

ARG uid
ARG gid

RUN groupadd --gid $gid linuxppc
RUN useradd --uid $uid --gid $gid linuxppc
RUN useradd linuxppc
USER linuxppc

COPY scripts/container-build.sh /bin/container-build.sh
Expand Down
7 changes: 1 addition & 6 deletions build/korg/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,7 @@ RUN curl -sSL $base_url/$tar_file > $tar_file && \
COPY korg/make-links.sh /tmp/make-links.sh
RUN /tmp/make-links.sh ${compiler_version} && rm /tmp/make-links.sh

ARG uid
ARG gid

RUN userdel -r ubuntu || true
RUN groupadd --gid $gid linuxppc
RUN useradd --uid $uid --gid $gid linuxppc
RUN useradd linuxppc
USER linuxppc

COPY scripts/container-build.sh /bin/container-build.sh
Expand Down
1 change: 1 addition & 0 deletions build/scripts/container-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ if [[ "$1" == "kernel" ]]; then
rc=$?
if [[ $rc -eq 0 ]]; then
tar -cjf /output/modules.tar.bz2 -C $mod_path lib
tar -czf /output/modules.tar.gz -C $mod_path lib
fi
else
echo "## Modules not configured"
Expand Down
10 changes: 0 additions & 10 deletions build/scripts/image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -117,23 +117,13 @@ if [[ -n "$https_proxy" ]]; then
cmd+="--build-arg https_proxy=$https_proxy "
fi

if [[ -z "$UID" ]]; then
UID=$(id -u)
fi

if [[ -z "$GID" ]]; then
GID=$(id -g)
fi

if [[ "$distro" == "korg" ]]; then
cmd+="--build-arg compiler_version=$version "
arch=$(uname -m)
cmd+="--build-arg base_url=https://mirrors.edge.kernel.org/pub/tools/crosstool/files/bin/${arch}/${version}/ "
cmd+="--build-arg tar_file=${arch}-gcc-${version}-nolibc-powerpc64-linux.tar.xz "
fi

cmd+="--build-arg uid=$UID "
cmd+="--build-arg gid=$GID "
cmd+="--build-arg from=$from "
cmd+="--build-arg apt_mirror=$APT_MIRROR "
cmd+="-t $image ."
Expand Down
2 changes: 1 addition & 1 deletion build/scripts/prune.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ case "$task" in
mkdir -p "artifacts"
for path in .config vmlinux System.map arch/powerpc/boot/zImage include/config/kernel.release \
arch/powerpc/kernel/asm-offsets.s arch/powerpc/boot/uImage modules.tar.bz2 \
sparse.log log.txt
modules.tar.gz sparse.log log.txt
do
if [[ -e "$path" ]]; then
mv "$path" artifacts/
Expand Down
10 changes: 3 additions & 7 deletions build/ubuntu/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ ARG apt_mirror
ENV apt_mirror=${apt_mirror}
RUN [ -n "$apt_mirror" ] && sed -i -e "s|ports.ubuntu.com|$apt_mirror|" /etc/apt/sources.list || true

# Deal with Ubuntu 21.10 repos moving to old-releases (ppc64le only)
RUN if echo "$from" | grep -q ".*ubuntu:21.10"; then sed -i -e "s|ports.ubuntu.com/ubuntu-ports|old-releases.ubuntu.com/ubuntu|" /etc/apt/sources.list; fi
# Deal with Ubuntu 21/22.10 repos moving to old-releases (ppc64le only)
RUN if echo "$from" | grep -q ".*ubuntu:2[12]\.10"; then sed -i -e "s|ports.ubuntu.com/ubuntu-ports|old-releases.ubuntu.com/ubuntu|" /etc/apt/sources.list; fi

RUN apt-get -q -y update && \
DEBIAN_FRONTEND=noninteractive \
Expand Down Expand Up @@ -52,11 +52,7 @@ RUN apt-get -q -y update && \
COPY ubuntu/make-links.sh /tmp/make-links.sh
RUN /tmp/make-links.sh ${compiler_version} && rm /tmp/make-links.sh

ARG uid
ARG gid

RUN groupadd --gid $gid linuxppc
RUN useradd --uid $uid --gid $gid linuxppc
RUN useradd linuxppc
USER linuxppc

COPY scripts/container-build.sh /bin/container-build.sh
Expand Down
4 changes: 3 additions & 1 deletion build/ubuntu/packages.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@ if [[ "$machine" == "ppc64le" ]]; then
PACKAGES+=" libhugetlbfs-dev"
PACKAGES+=" libmnl-dev"
PACKAGES+=" libmount-dev"
PACKAGES+=" liburing-dev"
if [[ "$major" -gt 20 ]]; then
PACKAGES+=" liburing-dev"
fi

# For the x86_64 container
if [[ "$major" -ge 21 ]]; then
Expand Down
101 changes: 101 additions & 0 deletions etc/configs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Shared config fragments etc.

guest_configs = [
'arch/powerpc/configs/guest.config',
'netconsole-n',
'ibmveth', # For PowerVM LPARs
'kuap',
'criu', # For seccomp tests
'user-ns', # For seccomp tests
'lkdtm', # Because it's useful
'ptdump', # Because it's useful
'strict-rwx', # Get some test coverage
'kfence',
'srr-debug',
'irq-soft-mask-debug',
'printk-index',
'debug-atomic-sleep',
'secure-boot',
'debug-vm',
'btrfs-y', # Needed for F39
'vfat-y', # Needed for F39
'zram', # Needed for F39
]

guest_configs_4k = guest_configs + ['4k-pages']
guest_configs_maxsmp = guest_configs + ['nr-cpus-8192']

legacy_guest_configs = [
'arch/powerpc/configs/guest.config',
'netconsole-n',
'ibmveth', # For PowerVM LPARs
'ibmehea', # Needed for Power7
'strict-rwx-off', # Bloats image too much for netboot to work w/128MB RMA
'nr-cpus-64', # Shrink kernel size
'ftrace-n', # Shrink kernel size
]

pmac32_configs = [
'pmaczilog',
'devtmpfs',
'debugfs',
'ptdump',
'debug-atomic-sleep',
'cgroups-y',
'arch/powerpc/configs/guest.config',
]

g5_configs = [
'pmaczilog',
'debugfs',
'ptdump',
'pstore',
'kvm-pr-y',
'agp-uninorth-y',
]

cell_configs = [
'cell',
'lockdep-y',
'debug-atomic-sleep',
'xmon-non-default',
]

powernv_configs = [
'tools/testing/selftests/ftrace/config',
'tools/testing/selftests/bpf/config',
'criu', # needed for selftests-seccomp
'igb', # Needed on some machines
'xfs-y', # Needed on some machines
'bridge-y', # Needed on some machines
'ahci-y', # Needed on some machines
'i40e-y', # Needed on some machines
'kvm-pr-m',
'strict-rwx',
'debug-atomic-sleep',
'selftests',
'pci-iov',
'page-poisoning-y',
'srr-debug', # Get some test coverage
'livepatch',
'secure-boot',
'zram',
'ptdump',
'amdgpu-y',
'drm-aspeed-y', # Aspeed DRM driver for /dev/fb0 on powernv machines
'fb-y', # Enable frame buffer for /dev/fb0 & alignment test
'xmon-non-default', # Better oopses in logs
'xmon-rw',
'selinux', # Avoid selinux relabeling on Fedora machines
'vfio-y', # Test coverage of VFIO
'cgroups-y', # So podman can run
'gup-test-y', # Enable selftest
]

powernv_lockdep_configs = powernv_configs + ['lockdep-y']

corenet64_configs = [
'debug-info-n',
'ppc64e-qemu',
'arch/powerpc/configs/guest.config',
]
2 changes: 2 additions & 0 deletions etc/configs/64k-pages.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
CONFIG_PPC_4K_PAGES=n
CONFIG_PPC_64K_PAGES=y
2 changes: 2 additions & 0 deletions etc/configs/agp-uninorth-y.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
CONFIG_AGP_UNINORTH=y
CONFIG_AGP=y
1 change: 1 addition & 0 deletions etc/configs/btf-n.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_DEBUG_INFO_BTF=n
1 change: 1 addition & 0 deletions etc/configs/btrfs-y.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_BTRFS_FS=y
9 changes: 9 additions & 0 deletions etc/configs/cgroups-y.config
Original file line number Diff line number Diff line change
@@ -1 +1,10 @@
CONFIG_CGROUPS=y
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_RDMA=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_MISC=y
CONFIG_CGROUP_DEBUG=y
CONFIG_NET_CLS_CGROUP=y
CONFIG_CGROUP_NET_PRIO=y
CONFIG_CGROUP_NET_CLASSID=y
1 change: 1 addition & 0 deletions etc/configs/gcc-plugins-n.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_GCC_PLUGINS=n
1 change: 1 addition & 0 deletions etc/configs/gup-test-y.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_GUP_TEST=y
1 change: 1 addition & 0 deletions etc/configs/i40e-y.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_I40E=y
1 change: 1 addition & 0 deletions etc/configs/nr-cpus-8192.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_NR_CPUS=8192
2 changes: 0 additions & 2 deletions etc/configs/strict-rwx.config
Original file line number Diff line number Diff line change
@@ -1,4 +1,2 @@
CONFIG_CRASH_DUMP=n
CONFIG_RELOCATABLE=n
CONFIG_STRICT_KERNEL_RWX=y
CONFIG_STRICT_MODULE_RWX=y
1 change: 1 addition & 0 deletions etc/configs/vfat-y.config
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
CONFIG_VFAT_FS=y
8 changes: 8 additions & 0 deletions etc/configs/vfio-y.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
CONFIG_VFIO=y
CONFIG_VFIO_GROUP=y
CONFIG_VFIO_CONTAINER=y
CONFIG_VFIO_IOMMU_SPAPR_TCE=y
CONFIG_VFIO_PCI_CORE=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI=y
1 change: 1 addition & 0 deletions etc/configs/zram.config
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
CONFIG_ZRAM=y
CONFIG_ZSMALLOC=y
CONFIG_CRYPTO_LZO=y
8 changes: 8 additions & 0 deletions etc/defaults.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
QEMU_VERSION = '9.0.2'

DEFAULT_OLD_IMAGE = '[email protected]'
DEFAULT_NEW_IMAGE = 'fedora'
DEFAULT_IMAGES = [DEFAULT_OLD_IMAGE, DEFAULT_NEW_IMAGE]

CLANG_IMAGES = ['fedora', 'ubuntu']
SPARSE_IMAGES = ['fedora', 'ubuntu']
Loading

0 comments on commit 5ee9aeb

Please sign in to comment.