Use response files for `ghc` invocations #9367

Gabriella439 · 2023-10-24T21:58:27Z

Before this change, cabal could fail with the following error message when building very large Haskell packages:

ghc: createProcess: posix_spawnp: resource exhausted (Argument list too long)

This is because when the number of modules or dependencies grows large enough, then the ghc command line can potentially exceed the ARG_MAX command line length limit.

However, ghc supports response files in order to work around these sorts of command line length limitations, so this change enables the use of those response files.

Note that this requires taking a special precaution to not pass RTS options to the response file because there's no way that ghc can support RTS options via the response file. The reason why is because the Haskell runtime processes these options (not ghc), so if you store the RTS options in the response file then ghc's command line parser won't know what to do with them.

This means that ghc commands can still potentially fail if the RTS options get long enough, but this is less likely to occur in practice since RTS options tend to be significantly smaller than non-RTS options.

This also requires skipping the response file if the first argument is --interactive. See the corresponding code comment which explains why in more detail.

Please read Github PR Conventions and then fill in one of these two templates.

Template Α: This PR modifies cabal behaviour

Include the following checklist in your PR:

Patches conform to the coding conventions.
Any changes that could be relevant to users have been recorded in the changelog.
The documentation has been updated, if necessary.
Manual QA notes have been included.

Bonus points for added automated tests!

Depends-on: #10292

Gabriella439 · 2023-10-24T22:01:56Z

Note that this has been tested internally at Mercury not just for cabal commands but also downstream tools (like ghcid and haskell-language-server), which is how we stumbled upon the hie-bios issue.

We ran into this issue when building our internal Haskell monolith (which has ~7000 Haskell modules), and we verified that this change fixes the problem without breaking anything else.

geekosaur

Can you run fourmolu on this (make style)? That check is failing.

Cabal/src/Distribution/Simple/Program/GHC.hs

Kleidukos

Thanks a lot @Gabriella439 for this contribution. 🙇

Kleidukos · 2023-10-26T17:46:31Z

@Gabriella439 There are still some failures: https://github.com/haskell/cabal/actions/runs/6643530708/job/18050770478?pr=9367#step:16:14024

Gabriella439 · 2023-10-27T20:57:25Z

I'm trying to reproduce the problem locally, but when I run ./validate.sh -s build && ./validate.sh -s lib-suite I get a whole bunch of tests being skipped with "SKIP no cabal-install" and I'm not sure how to fix that

Mikolaj · 2023-10-28T14:15:00Z

Would any of these help?

https://github.com/haskell/cabal/blob/master/CONTRIBUTING.md#running-tests
https://github.com/haskell/cabal/blob/master/cabal-testsuite/README.md

Cabal/src/Distribution/Simple/Program/GHC.hs

mpickering

Looks like a good direction of travel to me.

I want to block this being merged until there is a way to keep the response file around.

A very common workflow for me is to copy the command line which cabal invokes GHC with so that I can debug GHC issues without going through cabal, it seems this patch would regress that workflow.

Currently, `cabal repl` has a `--keep-temp-files` option, and `cabal.project` has a `keep-temp-files` option but it only effects Haddock builds. This patch adds `--keep-temp-files` to `CommonSetupFlags`, making it available to all commands. The expanded `--keep-temp-files` flag is used for the `cabal repl` command and Haddock builds (retaining compatibility with the previous behavior). The flag will also be used for response files; see haskell#9367.

9999years · 2024-09-03T19:49:21Z

The requested support for preserving the temp files ended up being much more complicated to implement than I thought it would be, and I just wanted to check with the maintainers to make sure I'm on the right track.

I've put the flag in SetupCommonOptions because those are already passed to the build functions that need them, and I've attempted to thread it through the conversion functions in D.C.ProjectConfig.Legacy, but I keep getting test failures from the value getting dropped somewhere in the round-trip conversion; this is with cabal test cabal-install:unit-tests --test-options="-p ProjectConfig":

projectConfigKeepTempFiles = -NoFlag +Flag False,

Am I missing something? Is there a simpler way of doing this?

9999years · 2024-09-10T00:09:16Z

Ready for review, but we should probably merge #10292 first for the --keep-temp-files changes.

geekosaur · 2024-09-10T00:21:49Z

Added a dependency so Mergify will hold off until it's merged.

9999years · 2024-11-20T00:05:11Z

Ready for review! #10292 is merged and I've added a release note.

Mikolaj · 2024-11-20T10:55:50Z

@mpickering: I'm guessing your comment is addressed? Would you like have a second look?

mergify · 2024-12-07T16:43:34Z

This pull request has been removed from the queue for the following reason: pull request branch update failed.

The pull request can't be updated.

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

geekosaur · 2024-12-07T18:01:39Z

@Gabriella439, since you made this PR from your work account, you must rebase it manually and use the merge+no rebase label because neither Mergify nor we are allowed to update it (no access to your employer's GitHub organization).

Before this change, `cabal` could fail with the following error message when building very large Haskell packages: ``` ghc: createProcess: posix_spawnp: resource exhausted (Argument list too long) ``` This is because when the number of modules or dependencies grows large enough, then the `ghc` command line can potentially exceed the `ARG_MAX` command line length limit. However, `ghc` supports response files in order to work around these sorts of command line length limitations, so this change enables the use of those response files. Note that this requires taking a special precaution to not pass RTS options to the response file because there's no way that `ghc` can support RTS options via the response file. The reason why is because the Haskell runtime processes these options (not `ghc`), so if you store the RTS options in the response file then `ghc`'s command line parser won't know what to do with them. This means that `ghc` commands can still potentially fail if the RTS options get long enough, but this is less likely to occur in practice since RTS options tend to be significantly smaller than non-RTS options. This also requires skipping the response file if the first argument is `--interactive`. See the corresponding code comment which explains why in more detail. Co-Authored-By: Gabriella Gonzales <[email protected]>

mergify · 2024-12-09T17:25:47Z

This pull request has been removed from the queue for the following reason: pull request dequeued.

Pull request #9367 has been dequeued. Pull request automatically merged by a merge action.

You should look at the reason for the failure and decide if the pull request needs to be fixed or if you want to requeue it.

If you want to requeue this pull request, you need to post a comment with the text: @mergifyio requeue

geekosaur requested changes Oct 24, 2023

View reviewed changes

Cabal/src/Distribution/Simple/Program/GHC.hs Outdated Show resolved Hide resolved

Gabriella439 force-pushed the gabriella/response_files_2 branch 2 times, most recently from bbe0bd2 to b689455 Compare October 24, 2023 23:10

geekosaur approved these changes Oct 24, 2023

View reviewed changes

Kleidukos requested changes Oct 25, 2023

View reviewed changes

Cabal/src/Distribution/Simple/Program/GHC.hs Outdated Show resolved Hide resolved

Cabal/src/Distribution/Simple/Program/GHC.hs Outdated Show resolved Hide resolved

Kleidukos approved these changes Oct 25, 2023

View reviewed changes

mpickering reviewed Nov 1, 2023

View reviewed changes

Cabal/src/Distribution/Simple/Program/GHC.hs Outdated Show resolved Hide resolved

mpickering reviewed Nov 1, 2023

View reviewed changes

Cabal/src/Distribution/Simple/Program/GHC.hs Outdated Show resolved Hide resolved

mpickering requested changes Nov 1, 2023

View reviewed changes

9999years force-pushed the gabriella/response_files_2 branch from 9e4dad2 to b0ef42d Compare August 29, 2024 21:21

9999years mentioned this pull request Aug 29, 2024

Expand and unify --keep-temp-files #10292

Merged

5 tasks

9999years force-pushed the gabriella/response_files_2 branch 6 times, most recently from eff868d to 6d2f9f3 Compare September 3, 2024 19:09

9999years force-pushed the gabriella/response_files_2 branch 7 times, most recently from e170a59 to 7d04b62 Compare September 5, 2024 16:36

9999years force-pushed the gabriella/response_files_2 branch 2 times, most recently from 07570f9 to 25d97f8 Compare September 6, 2024 19:57

9999years force-pushed the gabriella/response_files_2 branch from 25d97f8 to b021b34 Compare November 7, 2024 22:42

9999years force-pushed the gabriella/response_files_2 branch 2 times, most recently from 44b913b to 2f72704 Compare November 19, 2024 22:54

geekosaur approved these changes Nov 20, 2024

View reviewed changes

9999years requested a review from mpickering November 22, 2024 23:30

mpickering approved these changes Dec 5, 2024

View reviewed changes

mpickering added the merge me Tell Mergify Bot to merge label Dec 5, 2024

mergify bot added ready and waiting Mergify is waiting out the cooldown period merge delay passed Applied (usually by Mergify) when PR approved and received no updates for 2 days labels Dec 5, 2024

geekosaur removed the merge me Tell Mergify Bot to merge label Dec 7, 2024

9999years force-pushed the gabriella/response_files_2 branch from 2f72704 to 6549f2d Compare December 9, 2024 17:16

9999years added merge me Tell Mergify Bot to merge merge+no rebase labels Dec 9, 2024

mergify bot merged commit 949464d into haskell:master Dec 9, 2024
42 checks passed

9999years deleted the gabriella/response_files_2 branch December 9, 2024 17:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use response files for `ghc` invocations #9367

Use response files for `ghc` invocations #9367

Gabriella439 commented Oct 24, 2023 •

edited by ulysses4ever

Loading

Gabriella439 commented Oct 24, 2023

geekosaur left a comment

Kleidukos left a comment

Kleidukos commented Oct 26, 2023

Gabriella439 commented Oct 27, 2023 •

edited

Loading

Mikolaj commented Oct 28, 2023

mpickering left a comment

9999years commented Sep 3, 2024

9999years commented Sep 10, 2024

geekosaur commented Sep 10, 2024

9999years commented Nov 20, 2024

Mikolaj commented Nov 20, 2024

mergify bot commented Dec 7, 2024

geekosaur commented Dec 7, 2024

mergify bot commented Dec 9, 2024

Use response files for ghc invocations #9367

Use response files for ghc invocations #9367

Conversation

Gabriella439 commented Oct 24, 2023 • edited by ulysses4ever Loading

Gabriella439 commented Oct 24, 2023

geekosaur left a comment

Choose a reason for hiding this comment

Kleidukos left a comment

Choose a reason for hiding this comment

Kleidukos commented Oct 26, 2023

Gabriella439 commented Oct 27, 2023 • edited Loading

Mikolaj commented Oct 28, 2023

mpickering left a comment

Choose a reason for hiding this comment

9999years commented Sep 3, 2024

9999years commented Sep 10, 2024

geekosaur commented Sep 10, 2024

9999years commented Nov 20, 2024

Mikolaj commented Nov 20, 2024

mergify bot commented Dec 7, 2024

geekosaur commented Dec 7, 2024

mergify bot commented Dec 9, 2024

Use response files for `ghc` invocations #9367

Use response files for `ghc` invocations #9367

Gabriella439 commented Oct 24, 2023 •

edited by ulysses4ever

Loading

Gabriella439 commented Oct 27, 2023 •

edited

Loading