Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(export): use createWriteStream to export json #277

Merged
merged 7 commits into from
Feb 12, 2025
Merged

refactor(export): use createWriteStream to export json #277

Changes from 2 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4ccb371
perf(export): Speed up and reduce memory usage
D-Sketon Jan 30, 2025
423d781
Merge branch 'master' into refactor/exports
D-Sketon Jan 30, 2025
8842720
refactor: use stream
D-Sketon Jan 30, 2025
914ef26
Merge branch 'refactor/exports' of github.com:D-Sketon/warehouse into…
D-Sketon Jan 30, 2025
e91a90b
Update src/database.ts
D-Sketon Jan 31, 2025
94fe973
fix: test
D-Sketon Feb 1, 2025
0be39aa
fix: coverage
D-Sketon Feb 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 6 additions & 20 deletions src/database.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { parse as createJsonParseStream } from './lib/jsonstream';
import BluebirdPromise from 'bluebird';
import { writev, promises as fsPromises, createReadStream } from 'graceful-fs';
import { promises as fsPromises, createReadStream } from 'graceful-fs';
import { pipeline, Stream } from 'stream';
import Model from './model';
import Schema from './schema';
Expand All @@ -14,16 +14,6 @@ const pkg = require('../package.json');
const { open } = fsPromises;
const pipelineAsync = BluebirdPromise.promisify(pipeline) as unknown as (...args: Stream[]) => BluebirdPromise<unknown>;

let _writev: (handle: fsPromises.FileHandle, buffers: Buffer[]) => Promise<unknown>;

if (typeof writev === 'function') {
_writev = (handle, buffers) => handle.writev(buffers);
} else {
_writev = async (handle, buffers) => {
for (const buffer of buffers) await handle.write(buffer);
};
}

async function exportAsync(database: Database, path: string): Promise<void> {
const handle = await open(path, 'w');

Expand All @@ -44,16 +34,12 @@ async function exportAsync(database: Database, path: string): Promise<void> {

if (!models[key]) continue;

const buffers = [];

if (i) buffers.push(Buffer.from(',', 'ascii'));

buffers.push(Buffer.from(`"${key}":`));

buffers.push(Buffer.from(models[key]._export()));
await _writev(handle, buffers);
let prefix = '';
if (i) {
prefix = ',';
}
await handle.write(`${prefix}"${key}":${models[key]._export()}`);
Copy link
Member

@SukkaW SukkaW Jan 30, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we are targeting the modern Node.js versions now, maybe we could try fs.createWriteStream?

let p: Promise<unknown> | undefined;
const writeStream = fs.createWriteStream(filePath);

p = asyncWriteToStream(writeStream, prefix + '\n');
if (p) await p;

for (let i = 0; i < itemsLength; i++) {
  p = asyncWriteToStream(writeStream, items[i] + '\n');
  // eslint-disable-next-line no-await-in-loop -- stream high water mark
  if (p) await p;
}

// You can find `asyncWriteToStream` implementation at https://github.com/SukkaW/foxts/blob/a7f4e40434b1a46d58d5062b9df7713486f677b2/src/async-write-to-stream/index.ts#L17

}

// End models
await handle.write('}}');
} catch (e) {
Expand Down
Loading