feat: add initial crossplatform backup serialization [WPB-10575] #3121

vitorhugods · 2024-11-22T00:59:17Z

WPB-10575 Cross Platform Backup: Write common backup / restore library

PR Submission Checklist for internal contributors

The PR Title
- conforms to the style of semantic commits messages¹ supported in Wire's Github Workflow²
- contains a reference JIRA issue number like SQPIT-764
- answers the question: If merged, this PR will: ... ³
The PR Description
- is free of optional paragraphs and you have filled the relevant parts to the best of your ability

What's new in this PR?

Issues

This PR adds initial serialization/deserialization to the backup module.

Solutions

Using Protobuf, and a common implementation for JS and Non-JS.

The main difference between JS and Non-JS, is that:

JS doesn't have 64-bit integers (a.k.a. Long) for date/millis
JS/browser doesn't deal with "files". We can't write to paths. JS loads everything into memory and deals with byte arrays. Android and iOS need to be more careful with this and avoiding having everything in memory, so dealing with data streams is better.

Dependencies

Needs releases with:

Testing

Test Coverage

I have added automated test to this contribution

PR Post Merge Checklist for internal contributors

If any soft of configuration variable was introduced by this PR, it has been added to the relevant documents and the CI jobs have been updated.

References

https://sparkbox.com/foundry/semantic_commit_messages
https://github.com/wireapp/.github#usage
E.g. feat(conversation-list): Sort conversations by most emojis in the title #SQPIT-764.

This commit introduces a new cross-platform backup module to the project with support for iOS, Web, and Android. Additionally, it enables JavaScript multiplatform support in existing `kalium/network-model` and `kalium/data` projects.

This commit introduces a new multiplatform backup feature including export and import capabilities. It adds data classes, serialization, tests, and Gradle dependencies to support cross-platform backup data handling, including proper ProtoBuf serialization.

github-actions · 2024-11-22T01:11:10Z

Test Results

3 309 tests 3 202 ✅ 5m 1s ⏱️
566 suites 107 💤
566 files 0 ❌

Results for commit bc8ddeb.

♻️ This comment has been updated with latest results.

datadog-wireapp · 2024-11-22T01:36:48Z

Datadog Report

Branch report: feat/add-initial-crossplatform-backup-serialization
Commit report: 8227a80
Test service: kalium-jvm

✅ 0 Failed, 3202 Passed, 107 Skipped, 34.91s Total Time

…latform-backup-serialization

sonarqubecloud · 2024-11-22T14:10:51Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

codecov-commenter · 2024-11-22T14:46:57Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (epic/multiplatform-backup@1a57f5d). Learn more about missing BASE report.

Additional details and impacted files

@@                     Coverage Diff                      @@
##             epic/multiplatform-backup    #3121   +/-   ##
============================================================
  Coverage                             ?   54.03%           
============================================================
  Files                                ?     1246           
  Lines                                ?    36152           
  Branches                             ?     3656           
============================================================
  Hits                                 ?    19534           
  Misses                               ?    15207           
  Partials                             ?     1411

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a57f5d...bc8ddeb. Read the comment docs.

---- 🚨 Try these New Features:

Flaky Tests Detection - Detect and resolve failed and flaky tests
JS Bundle Analysis - Avoid shipping oversized bundles

vitorhugods · 2024-11-22T15:02:59Z

backup/src/commonMain/kotlin/com/wire/backup/MPBackup.kt

+
+import kotlin.js.JsExport
+
+@JsExport


@JsExport is needed so this class can be called from JS later.

vitorhugods · 2024-11-22T15:03:53Z

backup/src/commonMain/kotlin/com/wire/backup/data/BackupData.kt

+expect class BackupDateTime
+
+expect fun BackupDateTime(timestampMillis: Long): BackupDateTime
+expect fun BackupDateTime.toLongMilliseconds(): Long


This is because JS doesn't have 64-bit Integers (Long).
So we can use Long for iOS/Android, and a custom Date wrapper for JS.

vitorhugods · 2024-11-22T15:05:49Z

backup/src/commonMain/kotlin/com/wire/backup/dump/MPBackupExporter.kt

+    @ObjCName("add")
+    fun addUser(user: BackupUser) {
+        allUsers.add(user)
+    }
+
+    @ObjCName("add")
+    fun addConversation(conversation: BackupConversation) {
+        allConversations.add(conversation)
+    }
+
+    @ObjCName("add")
+    fun addMessage(message: BackupMessage) {
+        allMessages.add(message)
+    }


I was planning on having add for all three methods, only changing the parameter.

Unfortunately it doesn't work on JS, as it is not a typed language...

So at least when calling from Swift it will look like this:

backupExporter.add(user: BackupUser(...)) backupExporter.add(conversation: BackupConversation(...)) backupExporter.add(message: BackupMesage(...))

vitorhugods · 2024-11-22T15:06:21Z

backup/src/commonMain/kotlin/com/wire/backup/ingest/MPBackupMapper.kt

+                        BackupMessageContent.Text(proContent.value.content)
+                    }
+
+                    null -> TODO()


Done in the next PR:

feat: support other message types [WPB-10575] #3122

vitorhugods · 2024-11-22T15:07:08Z

backup/src/jsMain/kotlin/com/wire/backup/data/BackupDateTime.kt

+    override fun equals(other: Any?): Boolean {
+        if (this === other) return true
+        if (other == null || this::class.js != other::class.js) return false
+
+        other as BackupDateTime
+
+        return this.toLongMilliseconds() == other.toLongMilliseconds()
+    }


For some reason backupDateTime == backupDateTime doesn't work on JS, so I had to do it myself like so.

johnxnguyen · 2024-11-26T08:21:32Z

backup/src/commonMain/kotlin/com/wire/backup/data/BackupData.kt

+
+@JsExport
+sealed class BackupMessageContent {
+    data class Text(val text: String) : BackupMessageContent()


Text messages also contain link previews, mentions, and are possibly a reply to another message. Are we planning to include these in the first version?

We can expand later, like what's done in:

feat(backup): add more types - metadata assets (WPB-14589) #3128

johnxnguyen · 2024-11-27T15:27:10Z

backup/src/commonMain/kotlin/com/wire/backup/dump/MPBackupExporter.kt

+
+    @OptIn(ExperimentalStdlibApi::class)
+    @ShouldRefineInSwift // Hidden in Swift
+    fun serialize(): ByteArray {


Is this the method we call to produce the backup? The comment suggests it's hidden but it the POC it is visible but returns a KotlinByteArray which I'm guessing I would create a file with and present it to the user. Though I'm concerned about the size of this data and loading it all into memory at once.

I'm wondering if it would be possible for the client to pass a URL to which the library could write the backup file to. This could allow for more memory efficienct handling of the backup data.

Indeed :)
The goal is to implement the byte-reading part in the commonMain code, which will be used internally for all clients.

JS will only have this bytearray alternative.

Android and iOS will be able to pass paths that will stream and take paginated chunks of data in, without having to load the whole file into memory

This PR only handles initial serialization. There are more PRs incoming with:

Expanded serialization

Encryption/Decryption

Creating APIs specific for dealing with files on iOS and Android

Solidifying/tweaking the API

vitorhugods added 2 commits November 22, 2024 00:34

chore: create backup module

0690300

This commit introduces a new cross-platform backup module to the project with support for iOS, Web, and Android. Additionally, it enables JavaScript multiplatform support in existing `kalium/network-model` and `kalium/data` projects.

vitorhugods requested review from yamilmedina, Garzas and ohassine November 22, 2024 00:59

github-actions bot assigned vitorhugods Nov 22, 2024

github-actions bot added 👕 size: XL type: feature ✨ labels Nov 22, 2024

vitorhugods mentioned this pull request Nov 22, 2024

feat: support other message types [WPB-10575] #3122

Merged

6 tasks

Merge branch 'epic/multiplatform-backup' into feat/add-initial-crossp…

3a1dc1d

…latform-backup-serialization

vitorhugods marked this pull request as ready for review November 22, 2024 12:25

fix: date comparison and exception handling on JS

bc8ddeb

vitorhugods commented Nov 22, 2024

View reviewed changes

yamilmedina approved these changes Nov 22, 2024

View reviewed changes

johnxnguyen reviewed Nov 26, 2024

View reviewed changes

johnxnguyen reviewed Nov 27, 2024

View reviewed changes

echoes-hq bot added the echoes: product-roadmap/feature Work contributing to adding a new feature as part of the product roadmap. label Nov 28, 2024

vitorhugods merged commit ccb8193 into epic/multiplatform-backup Dec 2, 2024
20 checks passed

vitorhugods deleted the feat/add-initial-crossplatform-backup-serialization branch December 2, 2024 12:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add initial crossplatform backup serialization [WPB-10575] #3121

feat: add initial crossplatform backup serialization [WPB-10575] #3121

vitorhugods commented Nov 22, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 22, 2024 •

edited

Loading

datadog-wireapp bot commented Nov 22, 2024 •

edited

Loading

sonarqubecloud bot commented Nov 22, 2024

codecov-commenter commented Nov 22, 2024

vitorhugods Nov 22, 2024

vitorhugods Nov 22, 2024

vitorhugods Nov 22, 2024

vitorhugods Nov 22, 2024

vitorhugods Nov 22, 2024

johnxnguyen Nov 26, 2024

vitorhugods Nov 29, 2024

johnxnguyen Nov 27, 2024

vitorhugods Nov 29, 2024

feat: add initial crossplatform backup serialization [WPB-10575] #3121

feat: add initial crossplatform backup serialization [WPB-10575] #3121

Conversation

vitorhugods commented Nov 22, 2024 • edited by github-actions bot Loading

PR Submission Checklist for internal contributors

What's new in this PR?

Issues

Solutions

Dependencies

Testing

Test Coverage

PR Post Merge Checklist for internal contributors

References

github-actions bot commented Nov 22, 2024 • edited Loading

Test Results

datadog-wireapp bot commented Nov 22, 2024 • edited Loading

Datadog Report

sonarqubecloud bot commented Nov 22, 2024

Quality Gate passed

codecov-commenter commented Nov 22, 2024

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitorhugods commented Nov 22, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 22, 2024 •

edited

Loading

datadog-wireapp bot commented Nov 22, 2024 •

edited

Loading