Database Migration Library

[alexbrdn]

Changes in the Thrift data model sometimes produce incompatibilities between newer code and older data in existing couchdb instances. A library to facilitate automatic migration of the data to the new data model version should be developed.

So far, there were a few Python scripts developed for migrating data (see scripts/migrations), but there is no automation to that. It is necessary to manually start the correct scripts in the correct order. Also, these scripts introduce a dependency to Python, which is otherwise unnecessary.

The requirements for the new database migration library are:

• the library should first verify that the database has data model version information stored in it, otherwise initialize this version info
• each migration script should have a version information, designating what version the db model will be after the script is successfully executed. each migration script presupposes the db in the version previous to its version, i.e. script number 5 requires the db to be in version 4.
• the library should determine what migration scripts are available and start necessary migrations in the proper order. E.g. if the db is in version 7 and there are scripts 1 through 10 available, scripts 8, 9, and 10 should be executed in order.
• after each migration script is executed, its version should be stored in couchdb so that if the next script after it fails, the correct version information is preserved.
• it is yet an open question whether the migrations should be represented by Java classes, or there is a better way to script couchdb migrations without resorting to Python
• it is probably a good idea to make this library a separate project from sw360portal if possible, as it can be useful beyond sw360. This is not strictly necessary at first.
• migrations should be started automatically during sw360 deployment

[alexbrdn]

@heydenreich, @mcjaeger, @MaximilianHuber, @bs-jokri, you are invited to give your feedback on this

[bs-jokri]

Another idea might be to use tools like this one
https://github.com/haduart/flycouchdb

However, I haven't tested it yet.

[alexbrdn]

Granted, flycouchdb runs in the JVM, but it introduces a dependency on Clojure. Better than Python, I guess, but not by much.

I haven't found any CouchDB migration library written in Java, which would be ideal.

[maxhbr]

I am not sure, but isn't Clojure a compile time dependency and python + python-couchdb a runtime dependency?

I think runtime dependency are worse.

[alexbrdn]

from discussion: we will try flycouchdb first. If that works, we won't try to invent the bicycle anew.

[maxhbr]

@alex-evo Another alternative:

add to every thrift structure the filed Int dataTypeRevision with a default value corresponding to the current revision (starting with 1, old thrift structures without that value are treated as 0). This would allow us to write migration scripts in the following way:

for each JSON-object check which type and revision it has. Then apply all known migrations and write the current revision into the objet.

Problem: how to handle global changes / merge of objects / ...