Merge pull request #16 from de-nets/WhatsAppDB

Added WhatsApp DB import

CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 1.2.0
+
+- Added WhatsApp import from DB
+- Added import reactions to signal
+- Updates docs
+
 ## 1.1.0
 
 - Added Telegram messages import

README.md

@@ -5,7 +5,6 @@ Import messages from other apps like Telegram or WhatsApp to Signal.
 ## Caveats
 
 - Only tested on Android
-- No group text support
 - The commands were run on macOS
 
 ## Prerequisites
@@ -21,6 +20,7 @@ See: [Signal](docs/Signal.md)
 Import messages from:
 
 - [Telegram](docs/Telegram.md)
+- [WhatApp DB](docs/WhatApp_DB.md)
 - [WhatApp export](docs/WhatApp_Export.md)
 
 ### Available commands
@@ -37,15 +37,34 @@ bin/move_to_signal.dart \
 -o build/move_to_signal_$(uname -s)_$(uname -m)
 ```
 
+## Feature Map
+
+| Name                       | Telegram | WhatApp DB | WhatApp export |
+| :------------------------- | :------: | :--------: | :------------: |
+| All 1-on-1 text messages   |    ✅    |     ✅     |       ❌       |
+| Group chats                |    ❌    |     ❌     |       ❌       |
+| Original timestamps        |    ✅    |     ✅     |       ❌       |
+| Reactions (emoji)          |    ❌    |     ✅     |       ❌       |
+| Media (images/audio/links) |    ❌    |     ❌     |       ❌       |
+
 ## Known issues
 
-### Language based date time format in Whatsapp exports
+### Language based date time format in WhatsApp exports
+
+NOTE: This can be avoided by using the [WhatApp DB](docs/WhatApp_DB.md) import.
 
-Whatsapp exports have language based time format in export file and for Android without seconds. In my case 01/12/2023, 23:59  
+WhatsApp exports have language based time format in export file and for Android without seconds. In my case 01/12/2023, 23:59  
 The new WhatsApp macOS App has a more usable format [01.12.23, 23:59:42] for the same message as Android, but in my case only loads the last 3 years of chat history.
 
 Please open an issue with a small anonymized sample export and your device language. Or better open a PR with a fix. :)
 
+### Missing messages form WhatsApp exports
+
+NOTE: This can be avoided by using the [WhatApp DB](docs/WhatApp_DB.md) import.
+
+If a message was part of a sent image the message won't be in the export.
+I opened an issue with WhatsApp, but don't know if this will ever be fixed.
+
 ## Sponsor this project
 
 [![paypal](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif)](https://paypal.me/movetosignal/5)

bin/move_to_signal.dart

@@ -1,9 +1,10 @@
 import 'package:move_to_signal/import/signal.dart';
 import 'package:move_to_signal/source/telegram.dart';
-import 'package:move_to_signal/source/whats_app.dart';
+import 'package:move_to_signal/source/whats_app_db.dart';
+import 'package:move_to_signal/source/whats_app_export.dart';
 
 void main(List<String> arguments) {
-  String command = 'ImportWhatsApp';
+  String command = 'ImportWhatsAppExports';
   bool verbose = false;
 
   // Read all arguments
@@ -23,8 +24,14 @@ void main(List<String> arguments) {
       telegramImport.run(arguments);
 
       break;
-    case 'ImportWhatsApp':
-      final whatsAppImport = WhatsApp();
+    case 'ImportWhatsAppDb':
+      final whatsAppImportDb = WhatsAppDb();
+      whatsAppImportDb.verbose = verbose;
+      whatsAppImportDb.run(arguments);
+
+      break;
+    case 'ImportWhatsAppExports':
+      final whatsAppImport = WhatsAppExport();
       whatsAppImport.verbose = verbose;
       whatsAppImport.run(arguments);
 

docs/Commands.md

@@ -6,7 +6,8 @@
 
 --command=
     [ImportTelegram] For Telegram exports
-    [ImportWhatsApp] For WhatsApp exports (default)
+    [ImportWhatsAppDb] For WhatsApp db import (needs decrypted db with wa-crypt-tools)
+    [ImportWhatsAppExports] For WhatsApp exports (default)
     [SignalDecrypt] Just to decrypt Signal backup file
     [SignalEncrypt] Just to encrypt Signal backup file
 
@@ -35,6 +36,14 @@
     [Prepare] Prepare the import by extracting all conversations from telegramJson into separate files to review. (default)
     [Import] Imports the files from step [Prepare] into the Signal database.
 
---whatsappExports=
-    Path to the WhatsApp export .txt file
+--whatsAppDb=
+    Path to the WhatsApp msgstore.db file
+
+--whatsAppExports=
+    [ImportWhatsAppDb] Path where the WhatsApp .txt file are written
+    [ImportWhatsAppExports] Path to the WhatsApp export .txt files
+
+--whatsAppMode=
+    [Prepare] Prepare the import by extracting all conversations from msgstore.db into separate files to review. (default)
+    [Import] Imports the files from step [Prepare] into the Signal database.
 ```

docs/Install.md

@@ -3,4 +3,5 @@
 - Dependencies
   - [Dart](https://dart.dev/) to run from source
   - Download/Clone this git or download arm macOS binary
-  - [signalbackup-tools](https://github.com/bepaald/signalbackup-tools) De and Encrypt the Signal backup
+  - [signalbackup-tools](https://github.com/bepaald/signalbackup-tools) De- and Encrypt the Signal backup
+  - [wa-crypt-tools](https://github.com/ElDavoo/wa-crypt-tools) Decrypt WhatsApp backup (Only needed to import from db)

docs/WhatApp_DB.md

@@ -0,0 +1,92 @@
+# WhatApp DB
+
+Always start by creating a [Signal](docs/Signal.md) backup.
+
+1. Create WhatsApp backup, decrypt the msgstore.db.crypt15 with [wa-crypt-tools](https://github.com/ElDavoo/wa-crypt-tools) and copy the msgstore.db to the working folder.
+
+2. Run MoveToSignal in terminal for prepare the import
+
+   Mac arm64 binary
+
+   ```bash
+   cd path/to/working/folder/
+
+   path/to/MoveToSignal/move_to_signal_Darwin_arm64 \
+   --command=ImportWhatsAppDb \
+   --signalBackup=./signal-YYYY-MM-DD-HH-mm-ss.backup \
+   --signalBackupKey=123451234512345123451234512345 \
+   --signalPhoneNumber=+49123456789 \
+   --whatsAppDb="path/to/msgstore.db" \
+   --whatsAppExports=. \
+   --whatsAppMode=Prepare \
+   --verbose
+   ```
+
+   From source
+
+   ```bash
+   cd path/to/working/folder/
+
+   dart run path/to/MoveToSignal/bin/move_to_signal.dart \
+   --command=ImportWhatsAppDb \
+   --signalBackup=./signal-YYYY-MM-DD-HH-mm-ss.backup \
+   --signalBackupKey=123451234512345123451234512345 \
+   --signalPhoneNumber=+49123456789 \
+   --whatsAppDb="path/to/msgstore.db" \
+   --whatsAppExports=. \
+   --whatsAppMode=Prepare \
+   --verbose
+   ```
+
+   A new folder named WhatsAppExportsFolder will be created for the export files.
+   WhatsApp exports will be named eg: +4912345678-(Screen name if found).txt
+
+3. Rename exports
+
+   Please review the all .txt files and make sure to file names start with the contact phone number the user uses with Signal.
+   At this point you can also merge files into one, if a user had multiple WhatsApp identities.
+   Please delete all files you don't want to import.
+
+   All WhatsApp export files must be renamed like:  
+   contactPhoneNumber-Screen Name.txt
+
+   eg: +49123456789-Max ExampleName.txt
+
+   Only the phone number important for WhatsApp DB imports.
+   The phone number needs to in international format starting with + and must only contain numbers.
+
+4. Run MoveToSignal in terminal to import the prepared messages
+
+   Mac arm64 binary
+
+   ```bash
+   cd path/to/working/folder/
+
+   path/to/MoveToSignal/move_to_signal_Darwin_arm64 \
+   --command=ImportWhatsAppDb \
+   --signalBackup=./signal-YYYY-MM-DD-HH-mm-ss.backup \
+   --signalBackupKey=123451234512345123451234512345 \
+   --signalPhoneNumber=+49123456789 \
+   --whatsAppExports=. \
+   --whatsAppMode=Import \
+   --verbose
+   ```
+
+   From source
+
+   ```bash
+   cd path/to/working/folder/
+
+   dart run path/to/MoveToSignal/bin/move_to_signal.dart \
+   --command=ImportWhatsAppDb \
+   --signalBackup=./signal-YYYY-MM-DD-HH-mm-ss.backup \
+   --signalBackupKey=123451234512345123451234512345 \
+   --signalPhoneNumber=+49123456789 \
+   --whatsAppExports=. \
+   --whatsAppMode=Import \
+   --verbose
+   ```
+
+   Once done, a new Signal backup file is created, like: signal-signal-YYYY-MM-DD-HH-mm-ss.backup (new timestamp)
+
+5. Follow the "After importing all messages" steps from [Signal](docs/Signal.md)

docs/WhatApp_Export.md

@@ -39,7 +39,7 @@ Always start by creating a [Signal](docs/Signal.md) backup.
    --signalBackup=./signal-YYYY-MM-DD-HH-mm-ss.backup \
    --signalBackupKey=123451234512345123451234512345 \
    --signalPhoneNumber=+49123456789 \
-   --whatsappExports=./whatsapp \
+   --whatsAppExports=./whatsapp \
    --verbose
    ```
 
@@ -53,7 +53,7 @@ Always start by creating a [Signal](docs/Signal.md) backup.
    --signalBackup=./signal-YYYY-MM-DD-HH-mm-ss.backup \
    --signalBackupKey=123451234512345123451234512345 \
    --signalPhoneNumber=+49123456789 \
-   --whatsappExports=./whatsapp \
+   --whatsAppExports=./whatsapp \
    --verbose
    ```
 

lib/import/signal.dart

@@ -200,6 +200,15 @@ class Signal {
       'VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)',
     );
 
+    // Prepare a statement to run it multiple times:
+    final reactionImport = _database!.prepare(
+      'INSERT INTO reaction '
+      '('
+      'message_id, author_id, emoji, date_sent, date_received'
+      ') '
+      'VALUES (?, ?, ?, ?, ?)',
+    );
+
     // Import all messages
     var counter = 0;
     var step = (_signalMessages.length / 10).ceil();
@@ -251,6 +260,25 @@ class Signal {
         signalMessageImportSuccess = true;
       }
 
+      // Get last insert id as signalMessageId
+      int signalMessageId =
+          _database!.select('select last_insert_rowid()').first.columnAt(0);
+
+      // Import reactions
+      for (var reaction in signalMessage.reactions) {
+        if (reaction.sendTimestamp == null) {
+          continue;
+        }
+
+        reactionImport.execute([
+          signalMessageId,
+          reaction.authorId,
+          reaction.reaction,
+          reaction.sendTimestamp,
+          reaction.receivedTimestamp ?? reaction.sendTimestamp! + 500,
+        ]);
+      }
+
       if (!verbose) continue;
 
       counter++;

lib/model/signal_message.dart

@@ -1,3 +1,5 @@
+import 'package:move_to_signal/model/signal_reaction.dart';
+
 class SignalMessage {
   int messageDateTime = 0;
   int? dateSent;
@@ -18,8 +20,10 @@ class SignalMessage {
   int unidentified = 1;
   int reactionsLastSeen = -1;
   int notifiedTimestamp = 0;
+  List<SignalReaction> reactions = [];
 
-  Map<String, dynamic> toJson() => {
+  @override
+  String toString() => {
         "dateSent": dateSent,
         "dateReceived": dateReceived,
         "dateServer": dateServer,
@@ -38,11 +42,8 @@ class SignalMessage {
         "unidentified": unidentified,
         "reactionsLastSeen": reactionsLastSeen,
         "notifiedTimestamp": notifiedTimestamp,
-      };
-
-  @override
-  String toString() =>
-      '$dateSent|$dateReceived|$dateServer|$threadId|$fromRecipientId|$fromDeviceId|$toRecipientId|$type|$body|$read|$mType|$st|$receiptTimestamp|$hasDeliveryReceipt|$hasReadReceipt|$unidentified|$reactionsLastSeen|$notifiedTimestamp';
+        "reactions": reactions,
+      }.toString();
 
   void setReceived() {
     dateSent = messageDateTime;

lib/model/signal_reaction.dart

@@ -0,0 +1,18 @@
+import 'dart:convert';
+
+class SignalReaction {
+  int? authorId;
+  String? reaction;
+  bool? fromMe;
+  int? sendTimestamp;
+  int? receivedTimestamp;
+
+  @override
+  String toString() => {
+        '"authorId"': authorId,
+        '"reaction"': jsonEncode(reaction),
+        '"fromMe"': fromMe,
+        '"sendTimestamp"': sendTimestamp,
+        '"receivedTimestamp"': receivedTimestamp,
+      }.toString();
+}

lib/model/signal_thread.dart

@@ -5,14 +5,12 @@ class SignalThread {
   int snippetType = 10485780;
   int lastSeen = 0;
 
-  Map<String, dynamic> toJson() => {
+  @override
+  String toString() => {
         "_id": threadId,
         "date": date,
         "snippet": snippet,
         "snippetType": snippetType,
         "lastSeen": lastSeen,
-      };
-
-  @override
-  String toString() => '$threadId|$date|$snippet|$snippetType|$lastSeen';
+      }.toString();
 }

lib/model/whats_app_message.dart

@@ -0,0 +1,24 @@
+import 'dart:convert';
+
+import 'package:move_to_signal/model/whats_app_reaction.dart';
+
+class WhatsAppMessage {
+  int timestamp = 0;
+  int receivedTimestamp = 0;
+  int receiptServerTimestamp = 0;
+  bool fromMe = true;
+  int read = 0;
+  String text = '';
+  List<WhatsAppReaction> reactions = [];
+
+  @override
+  String toString() => {
+        '"timestamp"': timestamp,
+        '"receivedTimestamp"': receivedTimestamp,
+        '"receiptServerTimestamp"': receiptServerTimestamp,
+        '"fromMe"': fromMe,
+        '"read"': read,
+        '"text"': jsonEncode(text),
+        '"reactions"': reactions,
+      }.toString();
+}

lib/model/whats_app_reaction.dart

@@ -0,0 +1,16 @@
+import 'dart:convert';
+
+class WhatsAppReaction {
+  String? reaction;
+  bool? fromMe;
+  int? sendTimestamp;
+  int? receivedTimestamp;
+
+  @override
+  String toString() => {
+        '"reaction"': jsonEncode(reaction),
+        '"fromMe"': fromMe,
+        '"sendTimestamp"': sendTimestamp,
+        '"receivedTimestamp"': receivedTimestamp,
+      }.toString();
+}