Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(data:export): properly handle big JSON exports #1140

Merged
merged 1 commit into from
Dec 6, 2024

Conversation

cristiand391
Copy link
Member

@cristiand391 cristiand391 commented Dec 6, 2024

What does this PR do?

Updates bulk export utility to ensure it doesn't check if the JSON file where records will be written already exists.
For a few thousands of records, the API will still return all in one batch so NUTs were passing but if the job splits results in +1 batch then data export bulk --result-format json fails when trying to save the second batch because the FS flag instructs node to fail if the file already exists (which does, from the 1st batch that was saved).

repro:
query ScratchOrgInfo records in our na40 hub, there are ~1.6M records that get split in ~8 batches:

sf data export bulk -q 'select id,ExpirationDate from scratchorginfo' --result-format json --target-org na40 --output-file data.json --wait 20

you should get Error (EEXIST): EEXIST: file already exists, open 'data.json'. With this fix you should get a valid json, check its length with jq length data.json.

What issues does this PR fix or reference?

forcedotcom/cli#3138
@W-17380350@

@shetzel shetzel merged commit 0e4d220 into main Dec 6, 2024
30 checks passed
@shetzel shetzel deleted the cd/bulk-export-json branch December 6, 2024 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants