Auto merge of #5088 - rust-lang:gha, r=<try>

[WIP] Switch to GitHub Actions - Part 2 - From within

This is a continuation of #5071. This time from a branch inside the rust-lang/rust-clippy repo, not from my fork, since secrets are not available in PRs from forks.

Copying the description of #5071 to here:

Closes #4577

~~This is just an experiment. I don't think we have a consensus _if_ we should move away from travis/appveyor.~~ We have consensus: #5071 (comment)

~~GHA would let us run up to 20 concurrent jobs. Since we have 15 integration tests and 4 (linux, linux 32-bit, macos, windows) basic tests, we would be able to run everything concurrently.~~ The org has a limit of 60 jobs across the org, so we limit the matrix of the integration tests to 6 concurrent jobs.

~~Also IIUC we only have to build Clippy once for every initegration test and then only check the repos.~~ Nope, dependent jobs exist, but they won't keep the artifacts (not even the checkout).

TODO before merge:

- [x] Add `DEPLOY_KEY` secret to github repo
- [x] test deployment on test branch `gh-test`#
  - [x] Test normal deployment
  - [x] Test deployment no changes
  - [x] Test deployment of tag
- [x] talk with `@rust-lang/infra` for bors, `@rust-lang/infra` is good with the move (crater also uses GHA+bors)
- [x] ~~Get remark + clippy_dev check to work on pushes (https://github.community/t5/GitHub-Actions/~Builds-are-not-triggered-with-on-paths/m-p/44075; I contacted GH support about this) ~~That seems to start working again yesterday 🤔 Let's hope it keeps working.~~ Or not: df9be48. Now it works again: 723786a. I think we should wait, until this is reliable. It appears, that it doesn't work on force pushes (sometimes?): 5814142~~ We need to run the bors tests unconditionally anyway (47138d1) so it doesn't really matter.
- [ ] impl homu checks for GHA #5071 (comment) -- I prepared: flip1995/rust-central-station@f40230d. I'd suggest to first add GHA and keep the travis and appveyor checks for a few days and to remove them in a second pass. The bors dummy jobs are added in 1a83b7a and work as expected: #5088 (comment). I opened rust-lang/rust-central-station#578
- [x] ~Add GHA badge to Cargo.toml (blocked on rust-lang/crates.io # 1838)~ Added a FIXME in
2332b57
- [x] ~Maybe we should also wait until GHA supports yaml anchors. https://github.community/t5/GitHub-Actions/Support-for-YAML-anchors/td-p/30336/~ WIll probably not be implemented in the near future.
- [x] Add back travis + appveyor files for transition period (!)

changelog: none

.github/deploy.sh

@@ -1,20 +1,7 @@
 #!/bin/bash
 
-# Automatically deploy on gh-pages
-
 set -ex
 
-SOURCE_BRANCH="master"
-TARGET_BRANCH="gh-pages"
-
-# Save some useful information
-REPO=$(git config remote.origin.url)
-SSH_REPO=${REPO/https:\/\/github.com\//git@github.com:}
-SHA=$(git rev-parse --verify HEAD)
-
-# Clone the existing gh-pages for this repo into out/
-git clone --quiet --single-branch --branch "$TARGET_BRANCH" "$REPO" out
-
 echo "Removing the current docs for master"
 rm -rf out/master/ || exit 0
 
@@ -23,59 +10,34 @@ mkdir out/master/
 cp util/gh-pages/index.html out/master
 python ./util/export.py out/master/lints.json
 
-if [[ -n "$TRAVIS_TAG" ]]; then
-    echo "Save the doc for the current tag ($TRAVIS_TAG) and point current/ to it"
-    cp -r out/master "out/$TRAVIS_TAG"
-    rm -f out/current
-    ln -s "$TRAVIS_TAG" out/current
+if [[ -n $TAG_NAME ]]; then
+  echo "Save the doc for the current tag ($TAG_NAME) and point current/ to it"
+  cp -r out/master "out/$TAG_NAME"
+  rm -f out/current
+  ln -s "$TAG_NAME" out/current
 fi
 
 # Generate version index that is shown as root index page
 cp util/gh-pages/versions.html out/index.html
-pushd out
 
+cd out
 cat <<-EOF | python - > versions.json
 import os, json
 print json.dumps([
     dir for dir in os.listdir(".") if not dir.startswith(".") and os.path.isdir(dir)
 ])
 EOF
-popd
-
-# Pull requests and commits to other branches shouldn't try to deploy, just build to verify
-if [[ "$TRAVIS_PULL_REQUEST" != "false" ]] || [[ "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]]; then
-    # Tags should deploy
-    if [[ -z "$TRAVIS_TAG" ]]; then
-        echo "Generated, won't push"
-        exit 0
-    fi
-fi
 
 # Now let's go have some fun with the cloned repo
-cd out
-git config user.name "Travis CI"
-git config user.email "[email protected]"
+git config user.name "GHA CI"
+git config user.email "[email protected]"
 
 if git diff --exit-code --quiet; then
-    echo "No changes to the output on this push; exiting."
-    exit 0
+  echo "No changes to the output on this push; exiting."
+  exit 0
 fi
-cd -
 
-# Get the deploy key by using Travis's stored variables to decrypt deploy_key.enc
-ENCRYPTION_LABEL=e3a2d77100be
-ENCRYPTED_KEY_VAR="encrypted_${ENCRYPTION_LABEL}_key"
-ENCRYPTED_IV_VAR="encrypted_${ENCRYPTION_LABEL}_iv"
-ENCRYPTED_KEY=${!ENCRYPTED_KEY_VAR}
-ENCRYPTED_IV=${!ENCRYPTED_IV_VAR}
-openssl aes-256-cbc -K "$ENCRYPTED_KEY" -iv "$ENCRYPTED_IV" -in .github/deploy_key.enc -out .github/deploy_key -d
-chmod 600 .github/deploy_key
-eval "$(ssh-agent -s)"
-ssh-add .github/deploy_key
-
-cd out
 git add .
 git commit -m "Automatic deploy to GitHub Pages: ${SHA}"
 
-# Now that we're all set up, we can push.
 git push "$SSH_REPO" "$TARGET_BRANCH"

.github/driver.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+set -ex
+
+# Check sysroot handling
+sysroot=$(./target/debug/clippy-driver --print sysroot)
+test "$sysroot" = "$(rustc --print sysroot)"
+
+if [[ ${OS} == "Windows" ]]; then
+  desired_sysroot=C:/tmp
+else
+  desired_sysroot=/tmp
+fi
+sysroot=$(./target/debug/clippy-driver --sysroot $desired_sysroot --print sysroot)
+test "$sysroot" = $desired_sysroot
+
+sysroot=$(SYSROOT=$desired_sysroot ./target/debug/clippy-driver --print sysroot)
+test "$sysroot" = $desired_sysroot
+
+# Make sure this isn't set - clippy-driver should cope without it
+unset CARGO_MANIFEST_DIR
+
+# Run a lint and make sure it produces the expected output. It's also expected to exit with code 1
+# FIXME: How to match the clippy invocation in compile-test.rs?
+./target/debug/clippy-driver -Dwarnings -Aunused -Zui-testing --emit metadata --crate-type bin tests/ui/cstring.rs 2> cstring.stderr && exit 1
+sed -e "s,tests/ui,\$DIR," -e "/= help/d" cstring.stderr > normalized.stderr
+diff normalized.stderr tests/ui/cstring.stderr
+
+# TODO: CLIPPY_CONF_DIR / CARGO_MANIFEST_DIR

.github/workflows/clippy.yml

@@ -0,0 +1,98 @@
+name: Clippy Test
+
+on:
+  push:
+    # Ignore bors branches, since they are covered by `clippy_bors.yml`
+    branches-ignore: [auto, try]
+    # Don't run Clippy tests, when only textfiles were modified
+    paths-ignore:
+    - 'COPYRIGHT'
+    - 'LICENSE-*'
+    - '**.md'
+    - '**.txt'
+  pull_request:
+    # Don't run Clippy tests, when only textfiles were modified
+    paths-ignore:
+    - 'COPYRIGHT'
+    - 'LICENSE-*'
+    - '**.md'
+    - '**.txt'
+
+env:
+  RUST_BACKTRACE: 1
+  CARGO_TARGET_DIR: '${{ github.workspace }}/target'
+  NO_FMT_TEST: 1
+
+jobs:
+  base:
+    runs-on: ubuntu-latest
+
+    steps:
+    # Setup
+    - uses: rust-lang/simpleinfra/github-actions/cancel-outdated-builds@master
+      with:
+        github_token: "${{ secrets.github_token }}"
+
+    - name: rust-toolchain
+      uses: actions-rs/[email protected]
+      with:
+        toolchain: nightly
+        target: x86_64-unknown-linux-gnu
+        profile: minimal
+
+    - name: Checkout
+      uses: actions/[email protected]
+
+    - name: Run cargo update
+      run: cargo update
+
+    - name: Cache cargo dir
+      uses: actions/cache@v1
+      with:
+        path: ~/.cargo
+        key: ${{ runner.os }}-x86_64-unknown-linux-gnu-${{ hashFiles('Cargo.lock') }}
+        restore-keys: |
+          ${{ runner.os }}-x86_64-unknown-linux-gnu
+
+    - name: Master Toolchain Setup
+      run: bash setup-toolchain.sh
+
+    # Run
+    - name: Set LD_LIBRARY_PATH (Linux)
+      run: |
+        SYSROOT=$(rustc --print sysroot)
+        echo "::set-env name=LD_LIBRARY_PATH::${SYSROOT}/lib${LD_LIBRARY_PATH+:${LD_LIBRARY_PATH}}"
+
+    - name: Build
+      run: cargo build --features deny-warnings
+
+    - name: Test
+      run: cargo test --features deny-warnings
+
+    - name: Test clippy_lints
+      run: cargo test --features deny-warnings
+      working-directory: clippy_lints
+
+    - name: Test rustc_tools_util
+      run: cargo test --features deny-warnings
+      working-directory: rustc_tools_util
+
+    - name: Test clippy_dev
+      run: cargo test --features deny-warnings
+      working-directory: clippy_dev
+
+    - name: Test cargo-clippy
+      run: ../target/debug/cargo-clippy
+      working-directory: clippy_workspace_tests
+
+    - name: Test clippy-driver
+      run: bash .github/driver.sh
+      env:
+        OS: ${{ runner.os }}
+
+    # Cleanup
+    - name: Run cargo-cache --autoclean
+      run: |
+        cargo install cargo-cache --debug
+        find ~/.cargo/bin ! -type d -exec strip {} \;
+        cargo cache --autoclean