Read legacy app's database, and migrate data

[gnprice]

When we launch this app, it will go out as the new version of the Zulip mobile app on the Play Store and App Store. That means that users who already have the Zulip app installed, and upgrade, will get the new app replacing the legacy app.

At that point, if we don't do any special work to make something else happen, the user will find that the app has effectively forgotten their login to whatever Zulip server or servers they were using, as well as any of the local settings they had in the app. That's because the new app stores its data in a different place and format from the old app, and simply won't see the old data.

It'd be nice to instead make it so that the logins and settings carry over from the legacy app.

Implementation

This will involve looking carefully through the details of how the legacy app stores its data, in order to write logic in the new app to consume it. Here's an outline; details are to be found in the legacy app's code, and in the code of its dependencies.

1. We’ll need to read the SQLite database. This is straightforward — we're using SQLite anyway, for the new app's own database.
2. Then, on Android, we’ll decompress the data.
   • ​For this, we can always just take the relevant chunk of Kotlin code (which we wrote) and wrap it in a Flutter plugin instead of in a React Native native module.
   • ​Alternatively and perhaps cleaner, we can also transcribe that logic into Dart instead.
3. Then we’ll deserialize the JSON.
4. ​We’ll drop most of the data — the stuff that’s fundamentally a cache for data that lives on the server — just as we do for most migrations within the RN app.
5. ​We’ll apply any needed migrations from the history of the RN app. We can basically just transcribe the existing code to Dart.
6. ​Finally, we have data in the latest schema from the RN app. We’ll write it out in the new app’s own schema, and load it from there.

Then to be useful, this logic needs to be correct the very first time it runs on any given user's device. It therefore needs to be thoroughly tested.

Here's some optional bonus extra steps. We'll skip them, because they'd only be relevant for devices upgrading directly from a zulip-mobile release older than v27.183 — which was dated 2022-04-12, over two and a half years ago now. (I originally wrote these down when this new Flutter-based app was just a proposal, partly in order to make the point that we could do the migration very thoroughly. Copying them here now mostly for readers' entertainment.)

Steps for unnecessarily comprehensive migration

• For step 1 above, "read the SQLite database", add the following sub-steps:
  • ​The SQLite database is what we’ll find for any install that’s run a version released since v27.183, back in 2022-04. The relevant PR was: storage: Use SQLite to replace AsyncStorage zulip-mobile#5309
  • ​For older app versions, on Android, the data is simply in a SQLite database at a different file path.
    ​ * ​For older app versions, on iOS, the data is in a more ad-hoc format, because that’s what RN chose to do on iOS for its “AsyncStorage” feature. Still straightforward, though more annoying:
    • ​Specifically, the whole thing is JSON-serialized to a file somewhere; except that where the values are bigger than some threshold (a few kB?), they get written separately to their own files.
    • ​One option is always to just take the relevant chunk of Objective-C code — it’s a few hundred lines — and vendor it, wrapped in a Flutter plugin instead of in a React Native native module.
      ​ * Alternatively and perhaps cleaner, we can transcribe that logic into Dart.

Scope

The data that's relevant here — the data that the legacy app stores, and that we won't discard in step 4 above — is listed here (from src/boot/store.js in the legacy app):

export const storeKeys: $ReadOnlyArray<$Keys<GlobalState>> = [
  'migrations', 'accounts', 'drafts', 'outbox', 'settings',
];

Of those five rows in the database table, two are essential for this issue:

• migrations is needed for step 5 above.
• accounts corresponds to our Accounts table, or globalStore.accounts.

The other three correspond to features we may or may not have implemented at launch time:

• settings: Store some client settings #97
• outbox: Implement outbox system #133
• drafts: (We don't have an issue for this in the tracker. Not a priority.)

For any keys where the new app doesn't have a place for the data, we'll simply ignore them. The user can always adjust again the handful of settings the app stores; and as for outbox and drafts, those features' UX in the legacy app makes them not especially reliable anyway, so it's no great loss for that data not to carry over.