Database batch update API

[disinvite]

Updating the sqlite db using executemany and generator functions is a necessary evil for performance. This change abstracts away the implementation details and provides a much friendlier interface:

# Before:

self._db.bulk_recomp_insert(
    ((addr, {"name": values["name"]}) for addr, values in dataset.items()),
    upsert=True,
)
self._db.bulk_match(
    (
        (orig_addr, recomp_addr)
        for recomp_addr, orig_addr in orig_by_recomp.items()
    )
)


# After:

with self._db.batch() as batch:
    # Assume we are in a loop
        batch.set_recomp(recomp_addr, name="Test")
        batch.match(orig_addr, recomp_addr)

You can use the batch object with the context manager as shown, or manually call batch.commit().

The batch object provides five methods. When you commit, the changes are applied in this order (regardless of what order the methods were called):

• insert_orig and insert_recomp will set an entity in the database, but only if there is no existing data. (i.e. INSERT.)
• set_orig and set_recomp will insert or update data for the entity. (i.e. INSERT or REPLACE.)
• match will combine an orig address with a recomp address if neither is already matched.

The other big change is that matching will now combine two entities if both the orig and recomp addresses already exist but neither are matched. Until now, it was only possible to match by setting an unused orig address on an existing recomp entity. This will finally allow us to have inserting data and matching entities as two distinct operations. I have new modularized match functions mostly done in the stash. We can add these incrementally and now finally have some good tests on the matching behavior.

I only used the batch in a few places in core.py for now. These new matching functions will use it extensively and we can refactor the existing code that works with thunks, vtordisps, and strings.

[madebr]

https://docs.python.org/3/reference/datamodel.html#object.exit

This will unconditionally commit on exit. Shouldn't it roll back, or cancel the operations when the exception argument is non-null?

[jonschz]

I also remember that there is some helper function and syntax trick involving yield to efficiently write these scopes. Can't remember from the top of my head, though

[madebr]

I also remember that there is some helper function and syntax trick involving yield to efficiently write these scopes. Can't remember from the top of my head, though

You're probably thinking about contextlib.contextmanager (or similar functions in that module)

[disinvite]

Yes, thank you. I added some basic handling to __exit__.

commit might be a poor name because the batch operation itself is only loosely atomic. Only the bulk_match function uses a SQL transaction because it requires two statements.

[disinvite]

I just added a SQL transaction to batch.commit() and some tests to demonstrate. I checked on my project branch (with upcoming changes that use this API more) and the performance difference of doing it all in a transaction seems negligible.

[jonschz]

I am rather busy at the moment. Not sure if I can review before Sunday. If everything works and @madebr hasn't found anything else, feel free to merge.