Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Slow metadata index operations when creating many objects #11

Open
tskisner opened this issue Aug 1, 2017 · 0 comments
Open

Slow metadata index operations when creating many objects #11

tskisner opened this issue Aug 1, 2017 · 0 comments
Assignees

Comments

@tskisner
Copy link
Member

tskisner commented Aug 1, 2017

The serial volume class uses an SQLite DB as its metadata store. When adding new objects to the volume, this DB is updated. Since we can't predict what the user is doing, each object insertion is a discrete SQL transaction. This takes a fraction of a second, but when first creating a volume and adding thousands of objects, this can be slow.

Note that this does not impact the MPI volume, since in that case metadata operations are stored locally in memory (very fast) and replayed to the main SQLite DB during a metadata sync using a single transaction (also very fast).

One solution is to also use an in-memory metadata store even in the serial case with explicit sync to disk.

@tskisner tskisner self-assigned this Aug 1, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant