Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSC2545: Image Packs (Emoticons & Stickers) #2545

Open
wants to merge 36 commits into
base: old_master
Choose a base branch
from

Conversation

Sorunome
Copy link
Contributor

@Sorunome Sorunome commented May 15, 2020

Rendered

Signed-off-by: Sorunome [email protected]

Author: @Sorunome

Future MSCs

Current implementations

Emote rendering (rendering of the <img> tag)

  • element-web
  • revolution (even handles data-mx-emoticon)
  • nheko
  • fluffychat (even handles data-mx-emoticon)
  • neochat
  • cinny

Sending, using the mentioned events here

  • revolution (emoticons)
  • fluffychat (emoticons, stickers)
  • neochat (emoticons, at least partial)
  • nheko (emoticons, stickers)
  • cinny (emoticons, stickers)
  • kazv (stickers)

Bridges

  • mx-puppet-discord
  • matrix-appservice-discord (partially, sending data-mx-emoticon only)

Implementation PRs

FluffyChat

Data model: https://gitlab.com/famedly/company/frontend/libraries/matrix_api_lite/-/merge_requests/26
SDK: https://gitlab.com/famedly/company/frontend/famedlysdk/-/merge_requests/726
Emoticons: https://gitlab.com/famedly/fluffychat/-/merge_requests/433
Stickers: https://gitlab.com/famedly/fluffychat/-/merge_requests/452

Nheko

Stickers: Nheko-Reborn/nheko#648
Sticker editor: Nheko-Reborn/nheko#669
Choosing emoticons: Nheko-Reborn/nheko@ea6b19b

Cinny

Emoticons and Stickers: cinnyapp/cinny#686

kazv

Creating and sending stickers: https://lily-is.land/kazv/kazv/-/merge_requests/71

proposals/2545-emotes.md Outdated Show resolved Hide resolved
@anoadragon453
Copy link
Member

People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance.

proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
@turt2live turt2live added kind:feature MSC for not-core and not-maintenance stuff proposal A matrix spec change proposal labels May 15, 2020
@turt2live turt2live self-requested a review May 15, 2020 14:32
@turt2live turt2live marked this pull request as draft May 15, 2020 14:32
@MTRNord
Copy link
Contributor

MTRNord commented May 15, 2020

People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance.

Fluffychat Android works too :)

Copy link
Member

@turt2live turt2live left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally seems to be in a sensible direction, though the duplicated effort for custom emoji is a bit bothersome :(

proposals/2545-emotes.md Outdated Show resolved Hide resolved
@MatMaul
Copy link

MatMaul commented May 20, 2020

What about adding a mandatory level of pack indirection ?

im.ponies.room_emotes would only contains something like:

{
  "packs": ["im.ponies.room_emotes.mypack1", "im.ponies.room_emotes.mypack2"]
}

and then use what is proposed here inside im.ponies.room_emotes.mypackX state events.

This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin.

@Sorunome
Copy link
Contributor Author

What about adding a mandatory level of pack indirection ?

im.ponies.room_emotes would only contains something like:

{
  "packs": ["im.ponies.room_emotes.mypack1", "im.ponies.room_emotes.mypack2"]
}

and then use what is proposed here inside im.ponies.room_emotes.mypackX state events.

This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin.

Not quite sure what you mean, here. That, for room emotes, you can specify other state keys that just extend that one pack?

While not explicitly stated in this MSC (yet?), if multiple packs have the same pack.short they are expected to be merged together - that way you can also overcome the 65k limit, however, when adding it to your account data you will have to add multiple entries for just a single pack. Maybe it would be better to explicitly add related state keys?

@MatMaul
Copy link

MatMaul commented May 20, 2020

Ok nevermind, I misread. I thought im.ponies.room_emotes was the state key, not the type.

proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
@theotheroracle
Copy link

how hard would it be to add versioning and uuids to emoji packs
like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

@Sorunome
Copy link
Contributor Author

Sorunome commented Jun 7, 2020

how hard would it be to add versioning and uuids to emoji packs
like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

You already have versioning with state events - or do you mean something like v1.0? It would be trivial to add more information like that, or a uuid, to the pack object.

That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms?

@theotheroracle
Copy link

theotheroracle commented Jun 7, 2020

how hard would it be to add versioning and uuids to emoji packs
like for example if i'm in a room with a sticker pack that matches the UUID and version of the pack i'm using, then it won't add those emotes to my emote list

You already have versioning with state events - or do you mean something like v1.0? It would be trivial to add more information like that, or a uuid, to the pack object.

That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms?

if emotes are added to a pack, then the version number should be changed, and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user, after thinking about it might make more sense to have a last modified date instead of a version number, and it just uses the newest one, because if you want to remove an emote from a pack, then it shouldn't add it back from an older version of the pack, and i don't see how any of this is affected by it being in a room or a user account

@Sorunome
Copy link
Contributor Author

Sorunome commented Jun 7, 2020

and i don't see how any of this is affected by it being in a room or a user account

emote_rooms is where you say that you want emotes in a room (by room id and by state key) visible everywhere for yourself. How to handle now, if you spread one pack over multiple state events by putting the same uuid? Do you have to enable all the state keys? Should it autodetermine? If the later, wouldn't that be complex for client development?

and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user

Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc.

@Sorunome
Copy link
Contributor Author

Sorunome commented Jun 7, 2020

Like, maybe the uuid and versioning stuff could be added in a separate msc building onto this - the goal of this was to keep things as simple as possible

@theotheroracle
Copy link

emote_rooms is where you say that you want emotes in a room (by room id and by state key) visible everywhere for yourself. How to handle now, if you spread one pack over multiple state events by putting the same uuid? Do you have to enable all the state keys? Should it autodetermine? If the later, wouldn't that be complex for client development?

ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.

Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc.

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

@Sorunome
Copy link
Contributor Author

Sorunome commented Jun 7, 2020

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

origin_server_ts can't be trusted and can easily be forged

ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.

What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set

@theotheroracle
Copy link

state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers

oh you could just put a modified tag in then with like unixtime or something

What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set

the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example

@Sorunome
Copy link
Contributor Author

Sorunome commented Jun 7, 2020

oh you could just put a modified tag in then with like unixtime or something

In a federated system there is no one true timesource. That is a mathematically unsolvable problem.

the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example

Updates would replace the previous state event. You don't need any "part" attribute for that. As for ordering, you could say they are ordered alphanumerically by the state key.

@z411
Copy link

z411 commented Jan 2, 2023

I've been trying emoji and sticker packs with both Nheko and Cinny and I've seen a discrepancy with the usage of the "info" object (currently optional according to spec) in the image pack event which leads to some discrepancies when shown by clients or bridges, and I think it'd be prudent to clarify it somehow in the spec to achieve parity between clients.

When adding a sticker with Cinny, the info object of the sticker (mimetype, image size) is not saved in the image pack event. It's only sent when sending the actual sticker, in the m.sticker event. On the other hand, on Nheko, when adding a sticker the info of the sticker does get saved into the image pack event. See:

    "sticker_created_in_cinny": {
                "url": "mxc://redacted",
                "usage": [
                    "sticker",
                    "emoticon"
                ]
            },
    "sticker_created_in_nheko": {
                "info": {
                    "h": 169,
                    "mimetype": "image/jpeg",
                    "size": 55289,
                    "w": 223
                },
                "url": "mxc://redacted"
            },

Cinny doesn't need the info in the image pack event; it seems to generate the info object when sending m.sticker. Meanwhile Nheko uses the info objects of the image pack event of the room to send the sticker, and since Cinny doesn't save this information in the image pack event, Nheko sends stickers created in Cinny with a info object with empty values:

(Sticker created in Cinny sent by Nheko)
"info": {
            "h": 0,
            "mimetype": "",
            "size": 0,
            "thumbnail_info": {
                "h": 0,
                "mimetype": "",
                "size": 0,
                "w": 0
            },

This results in stickers being too big or bridges not knowing how to handle them due to the lack of mimetype.

My question is: what's the correct way to handle this? Should clients be mandated to include info in the image pack event and use that information when sending the sticker, or should they handle stickers with no info and generate it on the fly when sending it?

Related Nheko issue Nheko-Reborn/nheko#1255

@HarHarLinks
Copy link
Contributor

My question is: what's the correct way to handle this? Should clients be mandated to include info in the image pack event and use that information when sending the sticker, or should they handle stickers with no info and generate it on the fly when sending it?

If the info object is intentionally left optional, then it follows that sending clients are mandated to check for existing info and use it if it exists, and if it does not exist generate a proper info object from the actual image. In that sense I would consider this a bug in Nheko, it should do a sanity check on the data it sends in the info object.

This results in stickers being too big

How does (continuing your example) cinny decide what dimensions to use? Does it use just one single hardcoded size (at least for a single dimension, e.g. height)? Would it ignore the dimensions stored by nheko?

Then the question could be reprased: Do we want to support stickers/emoji of variable size which may diverge from the native image dimensions, i.e. coded in the info object? The current design allows for the same image to be available as either sticker or emoji. I don't think a single info object suffices in that case, unless we define that emoji should be displayed using an appropriate fixed size (i.e. match line height) which may even be applied on the receiving client's side, while stickers use the info object.

proposals/2545-emotes.md Outdated Show resolved Hide resolved
@maxwell-kalin

This comment was marked as off-topic.

Copy link
Member

@turt2live turt2live left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WIP review

proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
If the new `data-mx-emoticon` attribute has a value, it is ignored. Due to limitations of some libraries
the attribute may even look like `data-mx-emoticon=""`.

The `src` attribute *must* be a mxc url. Other URIs, such as `https`, `mailto` etc. are not allowed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `src` attribute *must* be a mxc url. Other URIs, such as `https`, `mailto` etc. are not allowed.
The `src` attribute *must* be a mxc uri. Other URIs, such as `https`, `mailto` etc. are not allowed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, we need consequences listed: does the event not get rendered, or just the emoticon?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've applied the suggestion in the first comment in a separate commit.

We need consequences listed: does the event not get rendered, or just the emoticon?

I don't see why the rest of the event should not be rendered. It'd be ideal to still be able to see the text in a message, even if the emotes aren't formatted properly (and are thus not rendered). Do you agree?

proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
Comment on lines +257 to +259
set one state event *per image*. While this might seem like a nice idea on the surface, it doesn't
scale well. There are people who easily use and want hundreds or even thousands of image packs accessible.
A simple dict of shortcode to mxc URI seems more appropriate for this.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think one could equally well argue the opposite. Given that matrix events are limited to 65536 bytes, and each image is going to be 1000 bytes or so (counting the URL and the imageinfo), you're not going to fit many images in an event.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a go at calculating a rough estimate of how many images (as defined by this MSC) you could fit into a single 65535 byte state event: https://github.com/anoadragon453/emote-estimate

The result was ~400 emotes per pack. This actually seems fairly tolerable for every day usage, especially as Discord has a maximum of 250 emotes per server (assuming you pay for that limit bump).

The main concern I can see is bridging; Slack seemingly lets you have infinite custom emoji per workspace, so a bridge would need to split that up into multiple packs, which may look a little ugly in the UI.

But my opinion is that 400 emotes is fine for now, and people are getting along with the current limits. A future MSC could add a method for merging image packs defined in separate state events together, but I don't think the current limit is a blocker.

3. Space image packs (defined in the hierarchy of canonical spaces for the current room)
4. Room image packs (defined in the currently open room's state)

Furthermore, clients are expected to de-duplicate images so that same images are not displayed multiple
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that de-duplicating images is the best UI. If there is an image that appears in two image packs, and I'm used to picking that image from pack 1, but the client only displays it in pack 2, then I won't be able to find the image that I want.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also makes it more difficult to manage packs, in case you want to manually remove duplicates.

Perhaps it would make more sense to only de-duplicate packs themselves?

Copy link

@Gremious Gremious Aug 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, less relevant and only marginally importnat, but assuming you can click on a sticker to view it's name in the chat like you can with discord stickers/emojis, that should probably be taken into account.

So if you have a smug/grimacing face in one pack titled "I'm gonna get u" and the same sticker in another pack titled "satisfied", you could potentially want to send the different stickers in different rooms, on the assumption that people are gonna click and read the name. And you'll probably search the sticker by the name you intend to convey.

Whether people actually use emojis/stickers like that is up to debate, but hey, I like naming my emoji nicely in discord, so I can kind of see it.


If de-duplicating is "automatic", then don't know if it's really even necessary. I don't see why a user shouldn't be able to just put 400 of the same sticker in their own pack if they wanted to. They probably won't anyway, buf if they do it as a joke, who cares. Feels like slightly wasted development time to me to put an arbitrary restriction.

A button to remove duplicates in the pack on demand sounds cool though, makes life easier if you just have a big sticker folder that might have a few repeating images, e.g. from downloading/collecting stickers you see manually.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A button to remove duplicates in the pack on demand sounds cool though, makes life easier if you just have a big sticker folder that might have a few repeating images, e.g. from downloading/collecting stickers you see manually.

Good idea, though this can be implemented in the client. The current spec won't get in the way of such a client feature.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deepbluev7 What are your thoughts on this, given your experience with implementing this MSC?

image pack event is in the room. This does not have to be unique from other packs in a room.
- `avatar_url`: (String, optional) The mxc uri of an avatar/icon to display for the pack. Defaults
to the room avatar, if the pack is in the room. Otherwise, the pack does not have an avatar.
- `usage`: (String[], optional) An array of the usages for this pack. Possible usages are `"emoticon"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if it's worth allowing implementations to create their own usages, in which case, we probably want the usages to be m.emoticon and m.sticker, and have the custom usages follow the Java package naming convention as usual.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the usual spec process of being able to add unstable prefixes to values in this field is sufficient? The current wording doesn't prevent that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. It just seems to me like naming it m.emoticon instead of just emoticon makes it easier and clearer. Anyways, it's a non-blocking comment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has anyone seen any use cases for additional "usages" in the wild while this MSC has been implemented? @deepbluev7 perhaps?

If not, perhaps it's just best to leave this as is.

@@ -0,0 +1,269 @@
# MSC2545: Image Packs (Emoticons & Stickers)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this MSC will intersect with MSC3911, I think we should consider this blocked until MSC3911 (or some other solution to the issues solved by that MSC) is merged.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this has to be blocked on MSC3911 - MSC2545 is already in use in the wild all over the place. MSC3911 will need to figure out how to address it, but not block the whole of this on media linking/auth.

@tusooa
Copy link
Contributor

tusooa commented Mar 9, 2024

Stickers support in kazv: https://lily-is.land/kazv/kazv/-/merge_requests/71

Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello all. I plan to breathe new life into this MSC by addressing any outstanding concerns and making changes as necessary.

I'm aware that this MSC has already been successfully implemented in numerous clients (thank you!), and I intend to remain compatible with those implementations as much as possible.

I very much look forward to seeing custom emotes and stickers become a part of the Matrix spec. Let's make it happen!

Comment on lines 200 to 201
For emoticons a client could add deliminators (e.g. `:`) around the image shortcode, meaning
that if an image has a shortcode of `emote`, the user can enter `:emote:` to send it. If there are
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a shortcode grammar defined anywhere? I see a definition of "must be a string starting and ending with a : character" and "must be less than 100 bytes" in MSC4027. But that seems like an odd MSC to be the one to officially define the grammar for shortcodes.

Perhaps this MSC could adopt the same rules, and MSC4027 reference it instead?

Comment on lines +257 to +259
set one state event *per image*. While this might seem like a nice idea on the surface, it doesn't
scale well. There are people who easily use and want hundreds or even thousands of image packs accessible.
A simple dict of shortcode to mxc URI seems more appropriate for this.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a go at calculating a rough estimate of how many images (as defined by this MSC) you could fit into a single 65535 byte state event: https://github.com/anoadragon453/emote-estimate

The result was ~400 emotes per pack. This actually seems fairly tolerable for every day usage, especially as Discord has a maximum of 250 emotes per server (assuming you pay for that limit bump).

The main concern I can see is bridging; Slack seemingly lets you have infinite custom emoji per workspace, so a bridge would need to split that up into multiple packs, which may look a little ugly in the UI.

But my opinion is that 400 emotes is fine for now, and people are getting along with the current limits. A future MSC could add a method for merging image packs defined in separate state events together, but I don't think the current limit is a blocker.

proposals/2545-emotes.md Outdated Show resolved Hide resolved
proposals/2545-emotes.md Outdated Show resolved Hide resolved
image pack event is in the room. This does not have to be unique from other packs in a room.
- `avatar_url`: (String, optional) The mxc uri of an avatar/icon to display for the pack. Defaults
to the room avatar, if the pack is in the room. Otherwise, the pack does not have an avatar.
- `usage`: (String[], optional) An array of the usages for this pack. Possible usages are `"emoticon"`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel the usual spec process of being able to add unstable prefixes to values in this field is sufficient? The current wording doesn't prevent that.

3. Space image packs (defined in the hierarchy of canonical spaces for the current room)
4. Room image packs (defined in the currently open room's state)

Furthermore, clients are expected to de-duplicate images so that same images are not displayed multiple
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It also makes it more difficult to manage packs, in case you want to manually remove duplicates.

Perhaps it would make more sense to only de-duplicate packs themselves?

@je-sendra
Copy link

Any updates on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocked Something needs to be done before action can be taken on this PR/issue. kind:feature MSC for not-core and not-maintenance stuff proposal A matrix spec change proposal
Projects
Status: Scheduled - v1.10
Development

Successfully merging this pull request may close these issues.