Problems when working with big diffs, such as vendored repository changes

[bruno-silva-cko]

Welcome

• Yes, I understand that the GitHub action repository is not the repository of golangci-lint itself.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).

Description of the problem

We use this workflow with only-new-issues: true for Pull Requests. This means developers don't need to tackle old linting issues as it would be quite hard and cumbersome. This has worked quite well til recently, when we vendored a few of our repositories due to other CI constraints.

When a dependency has many changes due to a major change or being a few version behind, golangci-lint ends up running for the whole repository instead of only the new changes. This is seemingly due to to a oktokit limitation. It is not properly documented, but it seems oktokit is only able to bring up to 300 files, otherwise it errors. An example can be seen below.

The potential source of this issue is this bit, which might need tweaking to consider another API (List pull requests files) + pagination.

Downloading binary https://github.com/golangci/golangci-lint/releases/download/v1.57.1/golangci-lint-1.57.1-linux-amd64.tar.gz ...
  failed to fetch pull request patch: RequestError [HttpError]: Sorry, the diff exceeded the maximum number of files (300). Consider using 'List pull requests files' API or locally cloning the repository instead.: {"resource":"PullRequest","field":"diff","code":"too_large"}
      at /home/runner/work/_actions/golangci/golangci-lint-action/v4/dist/run/index.js:47813:21
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async fetchPatch (/home/runner/work/_actions/golangci/golangci-lint-action/v4/dist/run/index.js:90861:27) {
    status: 406,
    response: {
      url: '[REDACTED]',
      status: 406,
      headers: {
        'access-control-allow-origin': '*',
        'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
        'content-length': '322',
        'content-security-policy': "default-src 'none'",
        'content-type': 'application/json; charset=utf-8',
        date: 'Thu, 04 Apr 20[24]([REDACTED]) 14:05:37 GMT',
        'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
        server: 'GitHub.com',
        'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
        vary: 'Accept-Encoding, Accept, X-Requested-With',
        'x-accepted-github-permissions': 'pull_requests=read; contents=read',
        'x-content-type-options': 'nosniff',
        'x-frame-options': 'deny',
        'x-github-api-version-selected': '2022-11-28',
        'x-github-media-type': 'github.v3; param=diff',
        'x-github-request-id': '4C00:3C94E7:44FF5[29]([REDACTED]):6F44646:660EB3B0',
        'x-ratelimit-limit': '15000',
        'x-ratelimit-remaining': '14982',
        'x-ratelimit-reset': '1712241910',
        'x-ratelimit-resource': 'core',
        'x-ratelimit-used': '18',
        'x-xss-protection': '0'
      },
      data: {
        message: "Sorry, the diff exceeded the maximum number of files ([30]([REDACTED])0). Consider using 'List pull requests files' API or locally cloning the repository instead.",
        errors: [Array],
        documentation_url: 'https://docs.github.com/rest/pulls/pulls#list-pull-requests-files'
      }
    },
    request: {
      method: 'GET',
      url: '[REDACTED]',
      headers: {
        accept: 'application/vnd.github.v3.diff',
        'user-agent': 'octokit-core.js/5.0.1 Node.js/20.8.1 (linux; x64)',
        authorization: 'token [REDACTED]'
      },
      request: {
        agent: [Agent],
        fetch: [Function: proxyFetch],
        hook: [Function: bound bound register]
      }
    }
  }

Version of golangci-lint

v1.57.1

Version of the GitHub Action

v4

Workflow file

name: golangci-lint

on:
  workflow_dispatch:
  workflow_call:
    inputs:
      golangci-lint-version:
        description: 'golang-ci linter version'
        default: 'v1.57.1'
        required: false
        type: string

jobs:
  runs:
    name: Golang CI Lint
    runs-on: ubuntu-20.04
    permissions:
      contents: read
      pull-requests: read
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 2 # required to make master build a cache

      - name: Check vendoring
        id: vendor
        run: |-
          [[ -d "vendor" ]] && echo "enabled=true" >> "$GITHUB_OUTPUT" || echo "enabled=false" >> "$GITHUB_OUTPUT"

      - name: Setup Golang
        uses: actions/setup-go@v5
        with:
          go-version-file: 'go.mod'
          check-latest: true
          cache: ${{ steps.vendor.outputs.enabled == 'false' }}

      - name: Download Go module dependencies
        if: ${{ steps.vendor.outputs.enabled == 'false' }}
        env:
          X_GITHUB_TOKEN: ${{ secrets.X_GITHUB_TOKEN }}
          X_GITHUB_USER: ${{ secrets.X_GITHUB_USER }}
        run: |-
          # Required to be able to download private go modules from git e.g. github.com/[REDACTED]/[REDACTED]
          git config --global url."https://${X_GITHUB_USER}:${X_GITHUB_TOKEN}@github.com".insteadOf "https://github.com"
          go mod download

      - name: Checkout tools repo
        run: |
          curl -s -O https://'${{secrets.X_GITHUB_TOKEN}}'@raw.githubusercontent.com/[REDACTED]/configs/master/go/.golangci.yaml

      - name: Lint feature branch
        uses: golangci/golangci-lint-action@v4
        if: github.ref != 'refs/heads/master' && github.ref != 'refs/heads/main'
        with:
          args: --out-${NO_FUTURE}format colored-line-number --config '.golangci.yaml' --max-issues-per-linter 0 --max-same-issues 0
          version: ${{ inputs.golangci-lint-version }}
          only-new-issues: true
          skip-pkg-cache: true

      # If this runs on master, it uses HEAD~ to check the linting differences, instead of trying
      # to create a patch (only works for PRs and it is available through only-new-issues).
      - name: Lint main branch and create scoped cache
        uses: golangci/golangci-lint-action@v4
        if: github.ref == 'refs/heads/master' || github.ref == 'refs/heads/main'
        with:
          args: --out-${NO_FUTURE}format colored-line-number --config '.golangci.yaml' --new-from-rev=HEAD~  --max-issues-per-linter 0 --max-same-issues 0
          version: ${{ inputs.golangci-lint-version }}
          skip-pkg-cache: true

Go version

1.21.8

Code example or link to a public repository

No Go code can be provided since is not a code issue, but more like PRs with high amount of diff changes

[ldez]

duplicate of #996