Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation #4

Merged
merged 108 commits into from
Jun 10, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
108 commits
Select commit Hold shift + click to select a range
17cd4ee
Initial implementation of MessagingModel
oxtoacart Mar 12, 2021
508e60e
Automated generation of one time preKeys
oxtoacart Mar 12, 2021
443d622
Progress on websocket client
oxtoacart Mar 13, 2021
6094a99
Authentication to real tassis over websockets works
oxtoacart Mar 13, 2021
b575f8a
Attempted with coroutines
oxtoacart Mar 16, 2021
5aaeb8c
Sort of working
oxtoacart Mar 16, 2021
fc6eb74
Test succeeding and completing
oxtoacart Mar 16, 2021
74539f5
More complete test, but failing because of invalid signature on devic…
oxtoacart Mar 16, 2021
d892719
Test passing
oxtoacart Mar 16, 2021
fa9061d
Test passing reliably
oxtoacart Mar 16, 2021
b986f33
Progress on modeling
oxtoacart Mar 19, 2021
bc7da8c
Revised datamodel, implemented linter suggestions, reasonably good ha…
oxtoacart Mar 19, 2021
c9a0779
Better factoring of MessagingTest
oxtoacart Mar 19, 2021
c57c74c
Updated to latest db and secrets libraries
oxtoacart Mar 19, 2021
7a37566
Ignore messages from senders who aren't in the Contact list
oxtoacart Mar 19, 2021
7951a5c
Immediately create Conversation when adding Contact
oxtoacart Mar 20, 2021
b5bc5af
Storing Base32 encoded versions of public and private identity keys i…
oxtoacart Mar 20, 2021
ab6fafb
More user friendly send API
oxtoacart Mar 21, 2021
b43c1f3
Removed ExperimentalTime dependency
oxtoacart Mar 21, 2021
234648d
Updated to latest libsignal dependencies
oxtoacart Mar 21, 2021
afed114
Only update conversation timestamp if new message time is more recent…
oxtoacart Mar 21, 2021
b541864
Upgraded to latest secrets and db
oxtoacart Mar 22, 2021
32a759f
Took out database contents logging in test
oxtoacart Mar 22, 2021
6f6b42c
Added ability to set my display name
oxtoacart Mar 22, 2021
9a1c801
Simplified contact/conversation model
oxtoacart Mar 23, 2021
d486531
Rationalized naming
oxtoacart Mar 23, 2021
a4f7676
Enabled lost connection checking for websockets
oxtoacart Mar 23, 2021
df6d3e2
Added ability to name database
oxtoacart Mar 24, 2021
a7f5b9e
Improved concurrency
oxtoacart Mar 25, 2021
6cc45f9
Cleaned up marking of failures
oxtoacart Mar 25, 2021
b850dfd
Less chatty database interaction
oxtoacart Mar 25, 2021
7bc6974
Startup and closing sequence cleanup
oxtoacart Mar 26, 2021
a1c1026
Better handling of websocket connect failures
oxtoacart Mar 26, 2021
68ee0f7
Moved websocket connection handling to workers
oxtoacart Mar 26, 2021
26972a1
Not locking in ClientWorker
oxtoacart Mar 26, 2021
f369e51
More connection management tweaks
oxtoacart Mar 26, 2021
b6b6c59
Allowing empty oneTimePreKeys
oxtoacart Mar 26, 2021
ed26c9e
More concurrency tweaks
oxtoacart Mar 27, 2021
640c579
Added request timeouts
oxtoacart Mar 27, 2021
f1f6c70
Added support for replyTo
oxtoacart Mar 27, 2021
f38cbb7
More connection management bug fixes, improved statusing for delivery…
oxtoacart Mar 27, 2021
8b3d8a3
Added attachment cipher from Signal-Android
oxtoacart Mar 31, 2021
af62007
Updated to latest tassis protocol buffer schema
oxtoacart Mar 31, 2021
e796c83
Attachments are sending
oxtoacart Apr 1, 2021
a87531c
More progress on uploads
oxtoacart Apr 1, 2021
fbeb6d2
Attachments work round-trip
oxtoacart Apr 1, 2021
604d0e0
Checking attachment size before encrypting
oxtoacart Apr 1, 2021
a903b2d
Added ability to include arbitrary metadata with attachments
oxtoacart Apr 1, 2021
d8bcfbd
Updated to latest db-android
oxtoacart Apr 3, 2021
2158116
Updated to latest db-android
oxtoacart Apr 3, 2021
915adce
Switched to using OkHhttp for websockets, tests now run on Android 22
oxtoacart Apr 4, 2021
5a9b045
Fixed close handling with OkHttp based websockets
oxtoacart Apr 4, 2021
50a4ae5
Fixed close handling with OkHttp based websockets
oxtoacart Apr 4, 2021
d45132a
More WebSocket connection tweaks
oxtoacart Apr 4, 2021
3a190b4
More websocket close handling improvements, more tests of websocket a…
oxtoacart Apr 5, 2021
42236fa
Upgraded db-android dependency
oxtoacart Apr 5, 2021
a5f7649
Capturing spam
oxtoacart Apr 6, 2021
716be98
Updated to latest db-android
oxtoacart Apr 6, 2021
7f5060b
Saving most recent attachment mime type on contact
oxtoacart Apr 7, 2021
1ff141f
Refactored model to accomodate new control message types
oxtoacart Apr 7, 2021
54de164
Progress towards handling control messages
oxtoacart Apr 7, 2021
e3354a9
More progress towards handling control messages
oxtoacart Apr 8, 2021
4166f8e
More progress towards handling control messages
oxtoacart Apr 8, 2021
78883fc
Local and global deletion of messages
oxtoacart Apr 8, 2021
8660b5a
Able to react to messages
oxtoacart Apr 8, 2021
3a198d3
Took auto delete out of model for now
oxtoacart Apr 8, 2021
93e12e5
Test refactoring
oxtoacart Apr 8, 2021
26205f6
More test refactoring
oxtoacart Apr 8, 2021
7c97610
Fixed attachments path building
oxtoacart Apr 8, 2021
952a0a1
Progress on disappearing messages
oxtoacart Apr 8, 2021
2e83b85
Disappearing messages work
oxtoacart Apr 9, 2021
c4dcc78
Recording viewed time
oxtoacart Apr 9, 2021
ce39b0d
Fixed disappearing messages test and only start disappearing timer lo…
oxtoacart Apr 9, 2021
9908c43
Fixed emoticon length
oxtoacart Apr 9, 2021
f60d230
Allowing 1 byte emoticons
oxtoacart Apr 9, 2021
b60c6d5
Setting contact createdTs correctly
oxtoacart Apr 9, 2021
29e46a9
Added ability to delete contacts
oxtoacart Apr 11, 2021
08f812d
Improved tests for initial disappear settings message
oxtoacart Apr 11, 2021
390cef9
Took out unnecessary checkin of Messages.java
oxtoacart Apr 12, 2021
40d8089
Logging cleanup
oxtoacart Apr 12, 2021
63a74a3
Allowing reactions to your own messages
oxtoacart Apr 12, 2021
6a35acc
Cleaning up public API
oxtoacart Apr 12, 2021
3fffd01
Cleaning up public API
oxtoacart Apr 12, 2021
60bc515
Updated README
oxtoacart Apr 14, 2021
8d882cb
Made db public again
oxtoacart Apr 14, 2021
3ecc5d9
Storing data within schema of supplied parent database
oxtoacart Apr 20, 2021
890f289
Added support for lazy attachments
oxtoacart Apr 21, 2021
0219d31
Using ktlint
oxtoacart Apr 23, 2021
1579c0a
Added auto-detection of mime type and thumbnails for images and videos
oxtoacart Apr 24, 2021
caa2fbc
Exporting secrets with API
oxtoacart Apr 25, 2021
089205c
Improved thumbnailing logic
oxtoacart Apr 25, 2021
50369cd
Don't recycle thumbnail bitmaps
oxtoacart Apr 25, 2021
816e475
Updated to latest secrets and db
oxtoacart Apr 26, 2021
4d9de90
Switched to using milliseconds instead of nanoseconds
oxtoacart Apr 26, 2021
99e9083
Added logic to delete orphaned attachments on startup
oxtoacart Apr 27, 2021
fc354d5
Added some JavaDoc comments
oxtoacart May 4, 2021
48db04a
Added introduction to README
oxtoacart May 11, 2021
8bb93a5
Updated to latest db-android
oxtoacart May 24, 2021
283a6e1
When remotely deleting messages, keep metadata around until user loca…
oxtoacart Jun 2, 2021
787d74d
Recording timestamp when message was deleted by sender
oxtoacart Jun 2, 2021
4bb3a30
Code review updates
oxtoacart Jun 8, 2021
10cdbb5
Updated to latest ktlint plugin
oxtoacart Jun 8, 2021
3976c36
Configured android flavor of ktlint
oxtoacart Jun 8, 2021
09ce412
Recording id of who remotely deleted a message
oxtoacart Jun 8, 2021
9131aac
Reordered some fields in the model
oxtoacart Jun 8, 2021
1d595a7
Using typed message for inbound messages
oxtoacart Jun 9, 2021
17a9642
Removed ide config
oxtoacart Jun 9, 2021
ba51ef3
Code review updates
oxtoacart Jun 9, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
messaging/src/androidTest/assets/image.jpg filter=lfs diff=lfs merge=lfs -text
15 changes: 15 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
*.iml
.gradle
/local.properties
/.idea/caches
/.idea/libraries
/.idea/modules.xml
/.idea/workspace.xml
/.idea/navEditor.xml
/.idea/assetWizardSettings.xml
.DS_Store
/build
/captures
.externalNativeBuild
.cxx
local.properties
674 changes: 674 additions & 0 deletions LICENSE

Large diffs are not rendered by default.

78 changes: 78 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# messaging-android
This library provides an API to facilitate secure end-to-end encrypted messaging on Android devices,
backed by a [fork of libsignal](https://github.com/getlantern/libsignal-protocol-java/) and
communicating via a [tassis](https://github.com/getlantern/tassis) messaging server.

messaging-android is currently used in [Lantern](https://lantern.io/) but is intended to be usable
by other parties in their own clients. tassis will eventually support federation, such that 3rd
parties can host their own back end and interoperate with other messaging clients.

## Protocol Buffers
messaging-android communicates with Tassis and internally stores data using protocol buffers.
Messages exchanged with tassis are defined in [Messages.proto](messaging/src/main/protos/Messages.proto),
which is just a copy of the protocol buffers defined in [tassis](https://github.com/getlantern/tassis/blob/main/model/Messages.proto).
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There must be a better way of sharing this model. Perhaps a repo which defines the shared models, even if it's currently only this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on the fence about this. I agree it's a little iffy to copy it, but OTOH a git submodule for just one file seems like a lot of work of its own.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a compromise of sorts: what if you added a note at the top of this file like:

Copy of getlantern/tassis/model/Messages.proto
Version: 39ca42e
If you update this file, please update this note as well.


The messaging-android data model is defined in [Model.proto](messaging/src/main/protos/Model.proto).

## Data Model
Copy link

@atavism atavism May 11, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we possibly add a brief overview to the top of the README instead of starting out with the data model? That this is an Android messaging library, Lantern Android integrates it, it re-uses code from Signal's protocol, etc?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Let me know if there's anything else you'd like to see in there.

messaging-android stores data in an [encrypted key-value store](https://github.com/getlantern/db-android/)
using keys that follow the below convention.

### /me (Model.Contact)
The contact entry for the user themselves.

### /contacts/d/[identityKey] (Model.Contact)
A direct contact, identified by their public IdentityKey

### /contacts/g/[groupId] (Model.Contact)
A group contact, identified by its group id

### /cba/[timestamp]/[d/[identityKey]|g/[groupId]]
An index of Contacts by most recent activity. A given Contact will appear only once in this index.

### /m/[senderIdentityKey]/[messageId] (Model.StoredMessage)
The full content of all Messages are stored here, including both sent and received messages.
The messageId is an id that's unique for messages sent from the given senderIdentityKey (in practice
it's a type 4 UUID).

### /cm/[d/[identityKey]|g/[groupId]]/[timestamp]/[senderIdentityKey]/[messageId]
An index of all messages for a given Contact, by the sent timestamp of the message.

### /dm/[disappearAt]/[senderIdentityKey]/[messageId]
An index of all messages that are supposed to auto disappear by some time (in unix milliseconds)

### /spam/[senderIdentityKey]/[timestamp]/[messageId] (Model.StoredMessage)
Messages that aren't worth showing to the user for one reason or another.

### /o/[timestamp]/[messageId] (Model.OutboundMessage)
A queue of outbound messages that are pending send. If sending to some recipients fails, messages
will be re-queued here for a limited period of time until they either send successfully or time
runs out.

### /ia/[senderIdentityKey]/[timestamp]/[messageId]/[attachmentId] (Model.InboundAttachment)
A queue of inbound attachments that are pending download. If downloading fails, downloads will be
re-queued here for a limited period of time until they either send successfully or time runs out.
hwh33 marked this conversation as resolved.
Show resolved Hide resolved

## Included Signal Code
The included Signal code (like AttachmentCipherInputStream) comes from https://github.com/signalapp/Signal-Android, not from https://github.com/signalapp/libsignal-service-java

## ktlint
This project is formatted and linted with ktlint using the [ktlint-gradle plugin](https://github.com/JLLeitschuh/ktlint-gradle).

You can install the [ktlint Intellij plugin](https://plugins.jetbrains.com/plugin/15057-ktlint-unofficial-)
for some support for linting within Android Studio.

### Add Commit Hook
./gradlew addKtlintCheckGitPreCommitHook

This adds a pre commit hook that lints all staged files upon commit.

### Manually Auto-format
./gradlew ktlintFormat

This auto-formats all Kotlin files in the project.

### Manually Check
./gradlew ktlintCheck

This manually runs the linter against all Kotlin files in the project.
41 changes: 41 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
// Top-level build file where you can add configuration options common to all sub-projects/modules.
buildscript {
ext.kotlin_version = "1.4.32"
ext.ktor_version = "1.5.2"

repositories {
google()
jcenter()
maven {
url "https://plugins.gradle.org/m2/"
}
}

dependencies {
classpath "com.android.tools.build:gradle:4.1.3"
classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
classpath("org.jlleitschuh.gradle:ktlint-gradle:10.1.0")

// NOTE: Do not place your application dependencies here; they belong
// in the individual module build.gradle files
}
}

allprojects {
repositories {
google()
jcenter()
mavenCentral()
maven { url "https://www.jitpack.io" }
}

apply plugin: "org.jlleitschuh.gradle.ktlint"

ktlint {
android = true
}
}

task clean(type: Delete) {
delete rootProject.buildDir
}
21 changes: 21 additions & 0 deletions gradle.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# Project-wide Gradle settings.
# IDE (e.g. Android Studio) users:
# Gradle settings configured through the IDE *will override*
# any settings specified in this file.
# For more details on how to configure your build environment visit
# http://www.gradle.org/docs/current/userguide/build_environment.html
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
org.gradle.jvmargs=-Xmx2048m -Dfile.encoding=UTF-8
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
# org.gradle.parallel=true
# AndroidX package structure to make it clearer which packages are bundled with the
# Android operating system, and which are packaged with your app"s APK
# https://developer.android.com/topic/libraries/support-library/androidx-rn
android.useAndroidX=true
# Automatically convert third-party libraries to use AndroidX
android.enableJetifier=true
# Kotlin code style for this project: "official" or "obsolete":
kotlin.code.style=official
Binary file added gradle/wrapper/gradle-wrapper.jar
Binary file not shown.
6 changes: 6 additions & 0 deletions gradle/wrapper/gradle-wrapper.properties
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#Wed Mar 10 20:46:29 CST 2021
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
distributionUrl=https\://services.gradle.org/distributions/gradle-6.5-all.zip
172 changes: 172 additions & 0 deletions gradlew
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
#!/usr/bin/env sh

##############################################################################
##
## Gradle start up script for UN*X
##
##############################################################################

# Attempt to set APP_HOME
# Resolve links: $0 may be a link
PRG="$0"
# Need this for relative symlinks.
while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`"/$link"
fi
done
SAVED="`pwd`"
cd "`dirname \"$PRG\"`/" >/dev/null
APP_HOME="`pwd -P`"
cd "$SAVED" >/dev/null

APP_NAME="Gradle"
APP_BASE_NAME=`basename "$0"`

# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS=""

# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD="maximum"

warn () {
echo "$*"
}

die () {
echo
echo "$*"
echo
exit 1
}

# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "`uname`" in
CYGWIN* )
cygwin=true
;;
Darwin* )
darwin=true
;;
MINGW* )
msys=true
;;
NONSTOP* )
nonstop=true
;;
esac

CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar

# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD="$JAVA_HOME/jre/sh/java"
else
JAVACMD="$JAVA_HOME/bin/java"
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD="java"
which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi

# Increase the maximum file descriptors if we can.
if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
MAX_FD_LIMIT=`ulimit -H -n`
if [ $? -eq 0 ] ; then
if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
MAX_FD="$MAX_FD_LIMIT"
fi
ulimit -n $MAX_FD
if [ $? -ne 0 ] ; then
warn "Could not set maximum file descriptor limit: $MAX_FD"
fi
else
warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
fi
fi

# For Darwin, add options to specify how the application appears in the dock
if $darwin; then
GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
fi

# For Cygwin, switch paths to Windows format before running java
if $cygwin ; then
APP_HOME=`cygpath --path --mixed "$APP_HOME"`
CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
JAVACMD=`cygpath --unix "$JAVACMD"`

# We build the pattern for arguments to be converted via cygpath
ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
SEP=""
for dir in $ROOTDIRSRAW ; do
ROOTDIRS="$ROOTDIRS$SEP$dir"
SEP="|"
done
OURCYGPATTERN="(^($ROOTDIRS))"
# Add a user-defined pattern to the cygpath arguments
if [ "$GRADLE_CYGPATTERN" != "" ] ; then
OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
fi
# Now convert the arguments - kludge to limit ourselves to /bin/sh
i=0
for arg in "$@" ; do
CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option

if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition
eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
else
eval `echo args$i`="\"$arg\""
fi
i=$((i+1))
done
case $i in
(0) set -- ;;
(1) set -- "$args0" ;;
(2) set -- "$args0" "$args1" ;;
(3) set -- "$args0" "$args1" "$args2" ;;
(4) set -- "$args0" "$args1" "$args2" "$args3" ;;
(5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
(6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
(7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
(8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
(9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
esac
fi

# Escape application args
save () {
for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
echo " "
}
APP_ARGS=$(save "$@")

# Collect all arguments for the java command, following the shell quoting and substitution rules
eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"

# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
cd "$(dirname "$0")"
fi

exec "$JAVACMD" "$@"
Loading