MSC2545: Image Packs (Emoticons & Stickers)

[Sorunome]

Rendered

Signed-off-by: Sorunome [email protected]

Author: @Sorunome

Future MSCs

• Image pack cross-room referencing / suggesting (MSC2545: Image Packs (Emoticons & Stickers) #2545 (comment))
• Additional metadata for stickers and emotes to find their pack (MSC2545: Image Packs (Emoticons & Stickers) #2545 (comment))
• Image pack ordering (MSC2545: Image Packs (Emoticons & Stickers) #2545 (comment))
• Multiple shortcodes for a single image (MSC2545: Image Packs (Emoticons & Stickers) #2545 (comment))
• E2EE support (MSC2545: Image Packs (Emoticons & Stickers) #2545 (comment))
• Image-Pack rooms (needing a room type) (MSC2545: Image Packs (Emoticons & Stickers) #2545 (comment))

Current implementations

Emote rendering (rendering of the <img> tag)

• element-web
• revolution (even handles data-mx-emoticon)
• nheko
• fluffychat (even handles data-mx-emoticon)
• neochat
• cinny

Sending, using the mentioned events here

• revolution (emoticons)
• fluffychat (emoticons, stickers)
• neochat (emoticons, at least partial)
• nheko (emoticons, stickers)
• cinny (emoticons, stickers)
• kazv (stickers)

Bridges

• mx-puppet-discord
• matrix-appservice-discord (partially, sending data-mx-emoticon only)

Implementation PRs

FluffyChat

Data model: https://gitlab.com/famedly/company/frontend/libraries/matrix_api_lite/-/merge_requests/26
SDK: https://gitlab.com/famedly/company/frontend/famedlysdk/-/merge_requests/726
Emoticons: https://gitlab.com/famedly/fluffychat/-/merge_requests/433
Stickers: https://gitlab.com/famedly/fluffychat/-/merge_requests/452

Nheko

Stickers: Nheko-Reborn/nheko#648
Sticker editor: Nheko-Reborn/nheko#669
Choosing emoticons: Nheko-Reborn/nheko@ea6b19b

Cinny

Emoticons and Stickers: cinnyapp/cinny#686

kazv

Creating and sending stickers: https://lily-is.land/kazv/kazv/-/merge_requests/71

[anoadragon453]

People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance.

[MTRNord]

People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance.

Fluffychat Android works too :)

[turt2live]

Generally seems to be in a sensible direction, though the duplicated effort for custom emoji is a bit bothersome :(

[MatMaul]

What about adding a mandatory level of pack indirection ?

im.ponies.room_emotes would only contains something like:

{
  "packs": ["im.ponies.room_emotes.mypack1", "im.ponies.room_emotes.mypack2"]
}

and then use what is proposed here inside im.ponies.room_emotes.mypackX state events.

This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin.

[Sorunome]

What about adding a mandatory level of pack indirection ?

im.ponies.room_emotes would only contains something like:

{
  "packs": ["im.ponies.room_emotes.mypack1", "im.ponies.room_emotes.mypack2"]
}

and then use what is proposed here inside im.ponies.room_emotes.mypackX state events.

This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin.

Not quite sure what you mean, here. That, for room emotes, you can specify other state keys that just extend that one pack?

While not explicitly stated in this MSC (yet?), if multiple packs have the same pack.short they are expected to be merged together - that way you can also overcome the 65k limit, however, when adding it to your account data you will have to add multiple entries for just a single pack. Maybe it would be better to explicitly add related state keys?

[MatMaul]

Ok nevermind, I misread. I thought im.ponies.room_emotes was the state key, not the type.

[theotheroracle]

how hard would it be to add versioning and uuids to emoji packs
like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

[Sorunome]

how hard would it be to add versioning and uuids to emoji packs
like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

You already have versioning with state events - or do you mean something like v1.0? It would be trivial to add more information like that, or a uuid, to the pack object.

That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms?

[theotheroracle]

how hard would it be to add versioning and uuids to emoji packs
like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

You already have versioning with state events - or do you mean something like v1.0? It would be trivial to add more information like that, or a uuid, to the pack object.

That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms?

if emotes are added to a pack, then the version number should be changed, and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user, after thinking about it might make more sense to have a last modified date instead of a version number, and it just uses the newest one, because if you want to remove an emote from a pack, then it shouldn't add it back from an older version of the pack, and i don't see how any of this is affected by it being in a room or a user account

[Sorunome]

and i don't see how any of this is affected by it being in a room or a user account

emote_rooms is where you say that you want emotes in a room (by room id and by state key) visible everywhere for yourself. How to handle now, if you spread one pack over multiple state events by putting the same uuid? Do you have to enable all the state keys? Should it autodetermine? If the later, wouldn't that be complex for client development?

and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user

Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc.

[Sorunome]

Like, maybe the uuid and versioning stuff could be added in a separate msc building onto this - the goal of this was to keep things as simple as possible

[theotheroracle]

emote_rooms is where you say that you want emotes in a room (by room id and by state key) visible everywhere for yourself. How to handle now, if you spread one pack over multiple state events by putting the same uuid? Do you have to enable all the state keys? Should it autodetermine? If the later, wouldn't that be complex for client development?

ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.

Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc.

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

[Sorunome]

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

origin_server_ts can't be trusted and can easily be forged

ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.

What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set

[theotheroracle]

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

oh you could just put a modified tag in then with like unixtime or something

What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set

the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example

[Sorunome]

oh you could just put a modified tag in then with like unixtime or something

In a federated system there is no one true timesource. That is a mathematically unsolvable problem.

the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example

Updates would replace the previous state event. You don't need any "part" attribute for that. As for ordering, you could say they are ordered alphanumerically by the state key.

[z411]

I've been trying emoji and sticker packs with both Nheko and Cinny and I've seen a discrepancy with the usage of the "info" object (currently optional according to spec) in the image pack event which leads to some discrepancies when shown by clients or bridges, and I think it'd be prudent to clarify it somehow in the spec to achieve parity between clients.

When adding a sticker with Cinny, the info object of the sticker (mimetype, image size) is not saved in the image pack event. It's only sent when sending the actual sticker, in the m.sticker event. On the other hand, on Nheko, when adding a sticker the info of the sticker does get saved into the image pack event. See:

    "sticker_created_in_cinny": {
                "url": "mxc://redacted",
                "usage": [
                    "sticker",
                    "emoticon"
                ]
            },
    "sticker_created_in_nheko": {
                "info": {
                    "h": 169,
                    "mimetype": "image/jpeg",
                    "size": 55289,
                    "w": 223
                },
                "url": "mxc://redacted"
            },

Cinny doesn't need the info in the image pack event; it seems to generate the info object when sending m.sticker. Meanwhile Nheko uses the info objects of the image pack event of the room to send the sticker, and since Cinny doesn't save this information in the image pack event, Nheko sends stickers created in Cinny with a info object with empty values:

(Sticker created in Cinny sent by Nheko)
"info": {
            "h": 0,
            "mimetype": "",
            "size": 0,
            "thumbnail_info": {
                "h": 0,
                "mimetype": "",
                "size": 0,
                "w": 0
            },

This results in stickers being too big or bridges not knowing how to handle them due to the lack of mimetype.

My question is: what's the correct way to handle this? Should clients be mandated to include info in the image pack event and use that information when sending the sticker, or should they handle stickers with no info and generate it on the fly when sending it?

Related Nheko issue Nheko-Reborn/nheko#1255

[HarHarLinks]

My question is: what's the correct way to handle this? Should clients be mandated to include info in the image pack event and use that information when sending the sticker, or should they handle stickers with no info and generate it on the fly when sending it?

If the info object is intentionally left optional, then it follows that sending clients are mandated to check for existing info and use it if it exists, and if it does not exist generate a proper info object from the actual image. In that sense I would consider this a bug in Nheko, it should do a sanity check on the data it sends in the info object.

This results in stickers being too big

How does (continuing your example) cinny decide what dimensions to use? Does it use just one single hardcoded size (at least for a single dimension, e.g. height)? Would it ignore the dimensions stored by nheko?

Then the question could be reprased: Do we want to support stickers/emoji of variable size which may diverge from the native image dimensions, i.e. coded in the info object? The current design allows for the same image to be available as either sticker or emoji. I don't think a single info object suffices in that case, unless we define that emoji should be displayed using an appropriate fixed size (i.e. match line height) which may even be applied on the receiving client's side, while stickers use the info object.

[turt2live]

WIP review

[turt2live]

Suggested change

	The `src` attribute *must* be a mxc url. Other URIs, such as `https`, `mailto` etc. are not allowed.
	The `src` attribute *must* be a mxc uri. Other URIs, such as `https`, `mailto` etc. are not allowed.

[turt2live]

also, we need consequences listed: does the event not get rendered, or just the emoticon?

[anoadragon453]

I've applied the suggestion in the first comment in a separate commit.

We need consequences listed: does the event not get rendered, or just the emoticon?

I don't see why the rest of the event should not be rendered. It'd be ideal to still be able to see the text in a message, even if the emotes aren't formatted properly (and are thus not rendered). Do you agree?

[richvdh]

I think one could equally well argue the opposite. Given that matrix events are limited to 65536 bytes, and each image is going to be 1000 bytes or so (counting the URL and the imageinfo), you're not going to fit many images in an event.

[anoadragon453]

I had a go at calculating a rough estimate of how many images (as defined by this MSC) you could fit into a single 65535 byte state event: https://github.com/anoadragon453/emote-estimate

The result was ~400 emotes per pack. This actually seems fairly tolerable for every day usage, especially as Discord has a maximum of 250 emotes per server (assuming you pay for that limit bump).

The main concern I can see is bridging; Slack seemingly lets you have infinite custom emoji per workspace, so a bridge would need to split that up into multiple packs, which may look a little ugly in the UI.

But my opinion is that 400 emotes is fine for now, and people are getting along with the current limits. A future MSC could add a method for merging image packs defined in separate state events together, but I don't think the current limit is a blocker.

[uhoreg]

I'm not sure that de-duplicating images is the best UI. If there is an image that appears in two image packs, and I'm used to picking that image from pack 1, but the client only displays it in pack 2, then I won't be able to find the image that I want.

[anoadragon453]

It also makes it more difficult to manage packs, in case you want to manually remove duplicates.

Perhaps it would make more sense to only de-duplicate packs themselves?

[Gremious]

Also, less relevant and only marginally importnat, but assuming you can click on a sticker to view it's name in the chat like you can with discord stickers/emojis, that should probably be taken into account.

So if you have a smug/grimacing face in one pack titled "I'm gonna get u" and the same sticker in another pack titled "satisfied", you could potentially want to send the different stickers in different rooms, on the assumption that people are gonna click and read the name. And you'll probably search the sticker by the name you intend to convey.

Whether people actually use emojis/stickers like that is up to debate, but hey, I like naming my emoji nicely in discord, so I can kind of see it.

---

If de-duplicating is "automatic", then don't know if it's really even necessary. I don't see why a user shouldn't be able to just put 400 of the same sticker in their own pack if they wanted to. They probably won't anyway, buf if they do it as a joke, who cares. Feels like slightly wasted development time to me to put an arbitrary restriction.

A button to remove duplicates in the pack on demand sounds cool though, makes life easier if you just have a big sticker folder that might have a few repeating images, e.g. from downloading/collecting stickers you see manually.

[anoadragon453]

A button to remove duplicates in the pack on demand sounds cool though, makes life easier if you just have a big sticker folder that might have a few repeating images, e.g. from downloading/collecting stickers you see manually.

Good idea, though this can be implemented in the client. The current spec won't get in the way of such a client feature.

[anoadragon453]

@deepbluev7 What are your thoughts on this, given your experience with implementing this MSC?

[uhoreg]

I wonder if it's worth allowing implementations to create their own usages, in which case, we probably want the usages to be m.emoticon and m.sticker, and have the custom usages follow the Java package naming convention as usual.

[anoadragon453]

I feel the usual spec process of being able to add unstable prefixes to values in this field is sufficient? The current wording doesn't prevent that.

[uhoreg]

Agreed. It just seems to me like naming it m.emoticon instead of just emoticon makes it easier and clearer. Anyways, it's a non-blocking comment.

[anoadragon453]

Has anyone seen any use cases for additional "usages" in the wild while this MSC has been implemented? @deepbluev7 perhaps?

If not, perhaps it's just best to leave this as is.

[richvdh]

Since this MSC will intersect with MSC3911, I think we should consider this blocked until MSC3911 (or some other solution to the issues solved by that MSC) is merged.

[ara4n]

I'm not sure this has to be blocked on MSC3911 - MSC2545 is already in use in the wild all over the place. MSC3911 will need to figure out how to address it, but not block the whole of this on media linking/auth.

[tusooa]

Stickers support in kazv: https://lily-is.land/kazv/kazv/-/merge_requests/71

[anoadragon453]

Hello all. I plan to breathe new life into this MSC by addressing any outstanding concerns and making changes as necessary.

I'm aware that this MSC has already been successfully implemented in numerous clients (thank you!), and I intend to remain compatible with those implementations as much as possible.

I very much look forward to seeing custom emotes and stickers become a part of the Matrix spec. Let's make it happen!

[anoadragon453]

Is there a shortcode grammar defined anywhere? I see a definition of "must be a string starting and ending with a : character" and "must be less than 100 bytes" in MSC4027. But that seems like an odd MSC to be the one to officially define the grammar for shortcodes.

Perhaps this MSC could adopt the same rules, and MSC4027 reference it instead?

[anoadragon453]

I had a go at calculating a rough estimate of how many images (as defined by this MSC) you could fit into a single 65535 byte state event: https://github.com/anoadragon453/emote-estimate

The result was ~400 emotes per pack. This actually seems fairly tolerable for every day usage, especially as Discord has a maximum of 250 emotes per server (assuming you pay for that limit bump).

The main concern I can see is bridging; Slack seemingly lets you have infinite custom emoji per workspace, so a bridge would need to split that up into multiple packs, which may look a little ugly in the UI.

But my opinion is that 400 emotes is fine for now, and people are getting along with the current limits. A future MSC could add a method for merging image packs defined in separate state events together, but I don't think the current limit is a blocker.

[anoadragon453]

I feel the usual spec process of being able to add unstable prefixes to values in this field is sufficient? The current wording doesn't prevent that.

[anoadragon453]

It also makes it more difficult to manage packs, in case you want to manually remove duplicates.

Perhaps it would make more sense to only de-duplicate packs themselves?

[je-sendra]

Any updates on this?