fix: use atomic writes for data store file operations #12715
+48
−5
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🦋 Changeset detectedLatest commit: 073ab1f The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
github-actions
bot
added
the
pkg: astro
CodSpeed Performance ReportMerging #12715 will not alter performanceComparing Summary
|
|
!preview atomic-writes |
|
Snapshots have been released for the following packages:
Publish LogBuild Log |
ematipico
reviewed
Dec 11, 2024
Comment on lines
16
to
23
| const dir = await fs.mkdtemp(tmpdir() + '/astro-'); | ||
| const tempFile = join(dir, 'temp'); | ||
| await fs.writeFile(tempFile, data); | ||
| try { | ||
| await fs.rename(tempFile, filePath); | ||
| } finally { | ||
| await fs.rmdir(dir).catch(() => {}); | ||
| } |
There was a problem hiding this comment.
I'm not too fond of this solution because now ALL projects, without discrimination, will incur in four (create temp dir, write file, rename file, remove directory) writing I/O operations instead of one. And writing operations are slow.
Maybe we could have an internal heuristic, where we enable this atomic rewriting only when we exceed a certain number of collections.
As you said, this is hard to test and incur in a project.
There was a problem hiding this comment.
This is not a high-volume write though. It's debounced, so only called once in builds. Switching based on collection size is a good idea though.
There was a problem hiding this comment.
I've done a new approach which is just a single write and a rename. Renames are cheap, so it shouldn't be much overhead compared to the current approach.
|
Ah, Windows isn't happy with this approach |
|
!preview atomic-writes |
|
Snapshots have been released for the following packages:
Publish LogBuild Log |
ematipico
approved these changes
Dec 11, 2024
|
So apparently this breaks differently. I'm seeing if I can reproduce that. |
|
!preview atomic-writes |
|
Snapshots have been released for the following packages:
Publish LogBuild Log |
ematipico
reviewed
Dec 12, 2024
ematipico
approved these changes
Dec 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
The data store has to write a number of different files to disk, which are in turn watched by the filesystem watcher. In some very large sites this was causing issues where the watcher would report the changed file before it had finished writing to disk.
This PR changes all data store file operations to use atomic writes: files are first written to a tmp dir, and then moved.
Fixes #12702
Testing
This is hard to reproduce. Tested manually using a reproduction with 50k pages and a deliberately shortened debounce time.
Docs