Implement sticker packs

[rocurley]

I'd like to add support for sticker packs to Riot. Since this would involve writing a lot of code, I wrote this little document up so we could discuss it before I've spent a long time on it. That said, while I'd love to have help, I'm happy to implement this myself.

Why sticker packs?

Most of my friends use Telegram as their chat client of choice. This isn't ideal for several reasons, and I'd like to get them to switch. It may seem silly, but I'm pretty sure the biggest lock-in factor there is stickers. Telegram's sticker system is really slick, and using anything without it or something similar feels like a real downgrade. Relatedly, I've framed most of this in comparison to Telegram's sticker system.

What makes sticker packs good?

I'd claim that the most important features are:

• User creatable: This is nice on several levels. It saves work on the dev side, obviously. But more than that, this lets people make stickers for their specific interests.

• Easily sendable: Stickers won't get used if it's not convenient to do so.

• Easily copyable: Almost all the stickers I use, I have because I saw one I liked, and I added it to my collection. Telegram makes this process trivial.

• Grouped: Part of the reason stickers spread so effectively is that if I see a sticker I like and add it to my collection, I get a bunch of other stickers along with it that I will likely enjoy.

The ideal outcome is an environment where stickers can be easily
created, and easily spread between people when they chat with each
other.

What are features we could do without?

I'm not super confident here, but:

• Updating packs. When the creator of a sticker pack adds new stickers to it, that change could be propagated out to users who have the pack, or not. I'm inclined towards not supporting that, mostly on implementation difficulty grounds. One could imagine having sticker packs label themselves as replacements for other ones, and make it easy to replace the old one if the user adds the new sticker pack, but that probably doesn’t need to be in the initial version.

Implementation

I’m proposing a specific implementation here as a proof of concept, but I’d be happy to see it improved. Mostly I’d like to get some discussion about this so I don’t spend a long time implementing this and then have it rejected.

The design goals with the implementation were:

• Work entirely within within matrix: no external servers.

• Avoid changes to the matrix protocol.

• Avoid requiring the homeserver to know about stickers.

• Avoid nonstandard messages.

This design achieves the first three. To allow subscribing to sticker packs, the fourth is compromised by adding a custom field to the image message when stickers are sent.

A sticker pack is an archive containing one or more images (stickers), and a metadata file. The metadata would associate each sticker with zero or more emoji, and contain the name of the sticker pack. It would also contain whatever information I forgot was important. Since it's likely we won't know ahead of time what information is important, we can version the spec and save the version in the metadata file. It’s unclear what file formats should be supported: that probably depends on how inline images shakes out (although this can be done with the standard image sharing until then).

To publish a sticker pack, one would:

• Create this archive (ideally assisted by the client, but it could start out as a manual process).

• Upload the sticker pack to your homeserver. The resulting MCX URI enables other users to subscribe (a misleading term, since it's impossible to update a sticker pack) to the sticker pack.

To subscribe to a sticker pack, the client would:

• Download the pack using the pack's URI.

• Save the URI in the client config. This makes subscribing to a sticker pack a a per-user, not per-device affair.

When using the pack, the client would:

• Check to make sure all the sticker packs in the client-config exist locally. This includes deleting sticker packs that have been unsubscribed from.

• Allow the user to easily browse and send stickers in the sticker packs.

• When sending a sticker, include in the message the sticker pack URI. This allows other users to preview and subscribe to other sticker packs.

• (This can be added late) If the user starts a message with an emoji, show all stickers indexed by that emoji.

Stages

I think the natural way to break this down into sub-tasks, each of which
can be implemented separately, is:

1. Implement sending (manually created) sticker packs from local storage. This is somewhat similar to Gregoor’s proposal for issue 2648. I’d be happy to coordinate with them on this.

2. (This doesn’t block anything else, it just has to come after 1) Implement a UI for creating sticker packs.

3. Synchronize a user’s sticker packs via the client-config.

4. When using a sticker from a pack, attach the link to the sticker pack.

5. Make a UI for subscribing to sticker packs you’ve seen.

[tessgadwa]

Let's do it.

See my comments at https://matrix.to/#/!blHeuKmYNNfIqDlWGO:matrix.org/$14844481771476529VOaQj:matrix.org

[rocurley]

Some details that tessgadwa and I hashed out over chat (it's all at the link above, so you can just read it there if you prefer).
Stickers should be 512x(<=512) vice versa (one dimension 512, one <=512) pngs, or vector graphics (svgs?).
Stickers can be associated with emoji. This implicitly requires the ability to recognize emoji. it seems simplest to piggyback off of the Emoji One conversion process (where if you get an emoji and nothing else in a message, it turns it into a bigger emoji).
We also discussed making some number of default sticker packs, which tessgadwa is interested in doing (in addition to UI work for steps 2 and 5). Stickerpacks of about 12 stickers, ideally including ones that convey: happy, bored, sad, angry, curious, celebratory, congratulatory, amused. (This was pretty off the cuff though).

[rocurley]

Some more discussion here. We're pretty much set for stage 1 as described above.

It was suggested that we find an existing sticker pack format and just use it. The only such thing I could find was apple's here. I'm sort of disinclined from using it, because it supports a bunch of features we probably don't want (animated stickers, for example), as well as not supporting some features it should (emoji). I think a simple json format along the lines of:

{
    "name" : "Goats with sunglasses"
    "author" : "Nunya Business"
    "stickerpack spec version" : 1.0
    "stickers" [
        {
            "file" : "evilGoat.svg"
            "emoji" : "😈🔥"
        },
        {
            "file" : "similingGoat.png"
            "emoji" : "😊"
        },
        {
            "file" : "nondescriptGoat.png"
            "emoji" : ""
        }
    ]
}

would be a lot easier.

We discussed where to store packs, with the tentative plan to use mxc for now. Other options include IPFS, some website, or torrent hashes. We probably want to have an external website for searching for sticker packs as well at some point.

[tessgadwa]

This works for me.

[ara4n]

how are the emoji fields meant to be used here?

[rocurley]

If the user starts a message with an emoji, show all stickers indexed by that emoji. Since stickers are used more or less in the same context as emoji, showing a relevant sticker when you type an emoji can be pretty useful

[tessgadwa]

I was also noticing that when I used the built-on Riot emojii, I only get 8 emojii showing up. Is this standard?

[tessgadwa]

Potential artist to contact suggested: http://www.deviantart.com/art/Phil-The-Owl-629856759

[tessgadwa]

Good discussion about in Riot main room today about image uploading vs sticker packs.

Goal is to allow users to access a common set of sticker packs direct from the app without having to upload their own each time from the hard drive.

[tessgadwa]

Another big goal: Create a way for artists to upload/share stickers under a CC license for inclusion in Riot.

[rocurley]

Currently thinking we should just base64 encode the images and stick them in the json file directly. It's a bit inefficient (4/3), but that way we don't have to add a dependency on some tar library or store every sticker separately on the server.

[ara4n]

@rocurley the disadvantage of doing that is that you don't end up with a canonical URL for the individual images, which will encourage clients to go and take the base64 out of the JSON and keep uploading them in duplicate to their HSes. Ideally the HS should deduplicate the content, but currently synapse doesn't sadly. Whereas if you reference the actual bits of artwork as separate mxc:// URLs, they will automatically get deduplicated. And clients won't have to mess around re-uploading when they send 'em?

[rocurley]

@ara4n That's definitely a benefit of storing them separately. My main concern there is individual stickers getting deleted. If you just have one file, that's less likely to happen. I still don't have a great model for how likely files are to be deleted though.

[rocurley]

Also worth figuring out: How do we cache stickerpacks locally? I was planning on fetching them from the homeserver at startup every time for now, but I don't love that option. I looked into localstorage, but 5Mb isn't very much.

[ara4n]

I'd just cache stickers using normal HTTP caching in the client (at least for riot-web); it should work fine :) In terms of content being deleted - there's always going to be a risk that the source content is deleted, whether it's a master copy or one of a million duplicates. For now I'd completely ignore it; practically speaking folks in Matrix hardly ever expire MXC content (and even if they do, it's only their remote caches), and in future switching to something like ipfs will make it go away anyway.

[rocurley]

Alright, sgtm.

[tessgadwa]

Aight.

…

On Fri, Jan 20, 2017 at 1:23 PM, Roger Curley ***@***.***> wrote: Alright, sgtm. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2950 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGLNG4ug22IJ86lrUNB3qND-p9ihCvA4ks5rUSXTgaJpZM4Ljr0n> .

-- Tess Gadwa Chief Executive Officer

_______________________________ Yes Exactly, Inc. | yesexactly.com | 413.325.8251

[ara4n]

one thought that could help here: if we switch to naming mxc urls based on content hashes then deduplication becomes trivial, as does fixing up and recognising content if the original content does get somehow deleted... see also https://github.com/matrix-org/matrix-doc/issues/446 and https://github.com/matrix-org/matrix-doc/issues/539

[ara4n]

@rocurley did this get anywhere?

[rocurley]

Sorry, haven't had much time on this for the past couple weeks. As it stands, you can send stickers if you've created and added them manually (with artisinal hand crafted http). Working on the UI for making sticker packs now.

[tessgadwa]

Cool. Let me know when the UI is ready.

[rocurley]

@ara4n so far we have code to send stickers from packs. The mechanisms for adding packs (making your own or adding packs you've seen) are still a work in progress. I'd like to get some code review before then, before we build too much on something that gets heavily modified. The issue is that it's not really mergeable at the moment: We'd be adding a sticker button that essentially nobody can use. It won't really be usable until we have the sticker pack creator working, or a set of sticker packs that are included by default (although that would be much easier to create if we had the sticker pack creator working). Idk what's the best solution here: I could make a mock PR?

[tessgadwa]

@rocurley where in the repo would one upload sample stickers to be used in testing?

[rocurley]

@tessgadwa Honnestly probably the easiest thing would be to send them over riot (to generate the mxc urls). That would make it pretty easy to put together a pack.

[tessgadwa]

Hey @rocurley I sent you some raw materials. Probably not what you're looking for but if you need specific image files there are plenty more under CC license (no attribution) on Google Images.

[aviraldg]

@tessgadwa Autocomplete is configured to only show the 8 most relevant ones, but there are more.

[tessgadwa]

@aviraldg how are we determining relevance?

[lukebarnard1]

@psyapathy It hasn't been implemented yet, with the main devs working on stabilising widgets and implementing communities.

[arpsyapathy]

@lukebarnard1 Sad... And then there is the implementation of the Emoji picker on click? To click on the button and get a full list of Emoji?

[lukebarnard1]

@psyapathy there's an issue for that somewhere. This isn't the place for discussion though, better off poking us in #riot

[ara4n]

So the original PR seems stuck, but we're hoping to get to this as part of Widgets in the relatively near future. Going to keep this bug open for historical value, and to act as a tracker for the upcoming stickerpack widget.

[Qwertie-]

I noticed there was some sticker picker integration in riot recently. Are there new updates to this issue?

[lboklin]

The integration seems to have been implemented in develop, but no actual sticker button appears when it's added.

[t3chguy]

Did you enable it in User Settings > Labs?

[lukebarnard1]

Stickers have landed as a "Labs" feature on the develop branch.

@lboklin you're better off asking in #riot:matrix.org or opening a new issue. But yes, as @t3chguy says, make sure you've enabled the feature flag in user settings.

[lboklin]

I had not! It works after enabling it there.

[lampholder]

Okay, so we now have stickers in Riot/Matrix :)

What's the delta (from a user's perspective) between our impl and this original discussion?

• User creatable: not trivially. Users can create stickers and send them to us to host, or host them on their own personal Dimension instance, but that's not something everybody is going to be capable of/interested in doing/willing to do
• Easily sendable: ✔️
• Easily copyable: not really. When you see a sticker somebody else has posted, there's not an easy link back to adding that stickerpack to your own collection
• Grouped: ✔️
• Updating packs: ✔️
  We get this 'for free' with the chosen impl (for adding new stickers, not modifying existing stickers in place)

[Qwertie-]

Is there any plan for user creatable packs? The way telegram does it is through a bot you message but I imagine creating a webUI or a custom page in riot for it might be easier to use.

[turt2live]

User-created (custom) sticker packs are in the proposal stage at the moment: matrix-org/matrix-spec-proposals#1256

[martindale]

I don't think this should be included in Riot by default, but rather made available as a plugin.

[schiessle]

It would be great if Matrix stickers would be compatible to Telegram stickers so that I can easily download a Telegram sticker pack and add it to Matrix.

[turt2live]

@schiessle keeping in mind that I'm the developer of the 3rd party project, Dimension (https://github.com/turt2live/matrix-dimension) allows you to import packs from Telegram for use in Riot/Matrix.

[dm17]

Would be great to be able to important all of the Telegram stickers easily. Switching from Telegram as they refuse to properly address why they won't release telegram-server.

[t3chguy]

https://github.com/turt2live/matrix-dimension supports importing telegram stickers by just pasting their pack link

[532910]

https://github.com/turt2live/matrix-dimension supports importing telegram stickers by just pasting their pack link

It looks like this is not true:
turt2live/matrix-dimension#363 (comment)

[t3chguy]

Moved to element-hq/element-meta#768