-
Notifications
You must be signed in to change notification settings - Fork 383
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MSC2545: Image Packs (Emoticons & Stickers) #2545
base: old_master
Are you sure you want to change the base?
Conversation
People wanting to try out an implementation of custom emotes on the web can use their existing homeserver account through this riot-web instance. |
Fluffychat Android works too :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally seems to be in a sensible direction, though the duplicated effort for custom emoji is a bit bothersome :(
What about adding a mandatory level of pack indirection ?
and then use what is proposed here inside This allows bundling and updating packs independently, and overcome the 65k limit by quite a margin. |
Not quite sure what you mean, here. That, for room emotes, you can specify other state keys that just extend that one pack? While not explicitly stated in this MSC (yet?), if multiple packs have the same |
Ok nevermind, I misread. I thought |
how hard would it be to add versioning and uuids to emoji packs |
You already have versioning with state events - or do you mean something like That being said, what if multiple packs have the same uuid but different emotes? Merge them? What if you spread one pack over multiple state events to overcome the 65k chars limit and then give them all the same uuid? Should they appear as one pack? How should that work with emote_rooms? |
if emotes are added to a pack, then the version number should be changed, and yeah i meant like a 1.0 thing, if multiple packs have the same uuid and the same version number, but different emotes, then the conflict should be reported to the user, after thinking about it might make more sense to have a last modified date instead of a version number, and it just uses the newest one, because if you want to remove an emote from a pack, then it shouldn't add it back from an older version of the pack, and i don't see how any of this is affected by it being in a room or a user account |
Would a user care about the version of an emote pack? How to determine which version is newer? Grammer for version numbers would need to be defined, etc. |
Like, maybe the uuid and versioning stuff could be added in a separate msc building onto this - the goal of this was to keep things as simple as possible |
ok if one pack was spread over multiple events, i would have them share the uuid and also have part numbers, like "part":1 "part":2 etc for A, mainting the order of emotes, and B, to prevent them from overlapping eachother.
state events have timestamps right, so you could just use the newest state event instead of bothering with version numbers |
origin_server_ts can't be trusted and can easily be forged
What would it need part for? It shouldn't matter if they are ordered correctly or not, emotes are an unordered set |
oh you could just put a modified tag in then with like unixtime or something
the most important reason for the part value is to distinguish continuations of the pack from updates and being able to order the emotes could matter to some pack maintainers, i would want to be able to do that for example |
In a federated system there is no one true timesource. That is a mathematically unsolvable problem.
Updates would replace the previous state event. You don't need any "part" attribute for that. As for ordering, you could say they are ordered alphanumerically by the state key. |
I've been trying emoji and sticker packs with both Nheko and Cinny and I've seen a discrepancy with the usage of the "info" object (currently optional according to spec) in the image pack event which leads to some discrepancies when shown by clients or bridges, and I think it'd be prudent to clarify it somehow in the spec to achieve parity between clients. When adding a sticker with Cinny, the
Cinny doesn't need the info in the image pack event; it seems to generate the
This results in stickers being too big or bridges not knowing how to handle them due to the lack of mimetype. My question is: what's the correct way to handle this? Should clients be mandated to include info in the image pack event and use that information when sending the sticker, or should they handle stickers with no info and generate it on the fly when sending it? Related Nheko issue Nheko-Reborn/nheko#1255 |
If the info object is intentionally left optional, then it follows that sending clients are mandated to check for existing info and use it if it exists, and if it does not exist generate a proper info object from the actual image. In that sense I would consider this a bug in Nheko, it should do a sanity check on the data it sends in the info object.
How does (continuing your example) cinny decide what dimensions to use? Does it use just one single hardcoded size (at least for a single dimension, e.g. height)? Would it ignore the dimensions stored by nheko? Then the question could be reprased: Do we want to support stickers/emoji of variable size which may diverge from the native image dimensions, i.e. coded in the info object? The current design allows for the same image to be available as either sticker or emoji. I don't think a single info object suffices in that case, unless we define that emoji should be displayed using an appropriate fixed size (i.e. match line height) which may even be applied on the receiving client's side, while stickers use the info object. |
This comment was marked as off-topic.
This comment was marked as off-topic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
WIP review
proposals/2545-emotes.md
Outdated
If the new `data-mx-emoticon` attribute has a value, it is ignored. Due to limitations of some libraries | ||
the attribute may even look like `data-mx-emoticon=""`. | ||
|
||
The `src` attribute *must* be a mxc url. Other URIs, such as `https`, `mailto` etc. are not allowed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The `src` attribute *must* be a mxc url. Other URIs, such as `https`, `mailto` etc. are not allowed. | |
The `src` attribute *must* be a mxc uri. Other URIs, such as `https`, `mailto` etc. are not allowed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also, we need consequences listed: does the event not get rendered, or just the emoticon?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've applied the suggestion in the first comment in a separate commit.
We need consequences listed: does the event not get rendered, or just the emoticon?
I don't see why the rest of the event should not be rendered. It'd be ideal to still be able to see the text in a message, even if the emotes aren't formatted properly (and are thus not rendered). Do you agree?
set one state event *per image*. While this might seem like a nice idea on the surface, it doesn't | ||
scale well. There are people who easily use and want hundreds or even thousands of image packs accessible. | ||
A simple dict of shortcode to mxc URI seems more appropriate for this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think one could equally well argue the opposite. Given that matrix events are limited to 65536 bytes, and each image is going to be 1000 bytes or so (counting the URL and the imageinfo), you're not going to fit many images in an event.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a go at calculating a rough estimate of how many images (as defined by this MSC) you could fit into a single 65535 byte state event: https://github.com/anoadragon453/emote-estimate
The result was ~400 emotes per pack. This actually seems fairly tolerable for every day usage, especially as Discord has a maximum of 250 emotes per server (assuming you pay for that limit bump).
The main concern I can see is bridging; Slack seemingly lets you have infinite custom emoji per workspace, so a bridge would need to split that up into multiple packs, which may look a little ugly in the UI.
But my opinion is that 400 emotes is fine for now, and people are getting along with the current limits. A future MSC could add a method for merging image packs defined in separate state events together, but I don't think the current limit is a blocker.
3. Space image packs (defined in the hierarchy of canonical spaces for the current room) | ||
4. Room image packs (defined in the currently open room's state) | ||
|
||
Furthermore, clients are expected to de-duplicate images so that same images are not displayed multiple |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure that de-duplicating images is the best UI. If there is an image that appears in two image packs, and I'm used to picking that image from pack 1, but the client only displays it in pack 2, then I won't be able to find the image that I want.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It also makes it more difficult to manage packs, in case you want to manually remove duplicates.
Perhaps it would make more sense to only de-duplicate packs themselves?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, less relevant and only marginally importnat, but assuming you can click on a sticker to view it's name in the chat like you can with discord stickers/emojis, that should probably be taken into account.
So if you have a smug/grimacing face in one pack titled "I'm gonna get u" and the same sticker in another pack titled "satisfied", you could potentially want to send the different stickers in different rooms, on the assumption that people are gonna click and read the name. And you'll probably search the sticker by the name you intend to convey.
Whether people actually use emojis/stickers like that is up to debate, but hey, I like naming my emoji nicely in discord, so I can kind of see it.
If de-duplicating is "automatic", then don't know if it's really even necessary. I don't see why a user shouldn't be able to just put 400 of the same sticker in their own pack if they wanted to. They probably won't anyway, buf if they do it as a joke, who cares. Feels like slightly wasted development time to me to put an arbitrary restriction.
A button to remove duplicates in the pack on demand sounds cool though, makes life easier if you just have a big sticker folder that might have a few repeating images, e.g. from downloading/collecting stickers you see manually.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A button to remove duplicates in the pack on demand sounds cool though, makes life easier if you just have a big sticker folder that might have a few repeating images, e.g. from downloading/collecting stickers you see manually.
Good idea, though this can be implemented in the client. The current spec won't get in the way of such a client feature.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@deepbluev7 What are your thoughts on this, given your experience with implementing this MSC?
image pack event is in the room. This does not have to be unique from other packs in a room. | ||
- `avatar_url`: (String, optional) The mxc uri of an avatar/icon to display for the pack. Defaults | ||
to the room avatar, if the pack is in the room. Otherwise, the pack does not have an avatar. | ||
- `usage`: (String[], optional) An array of the usages for this pack. Possible usages are `"emoticon"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if it's worth allowing implementations to create their own usages, in which case, we probably want the usages to be m.emoticon
and m.sticker
, and have the custom usages follow the Java package naming convention as usual.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel the usual spec process of being able to add unstable prefixes to values in this field is sufficient? The current wording doesn't prevent that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed. It just seems to me like naming it m.emoticon
instead of just emoticon
makes it easier and clearer. Anyways, it's a non-blocking comment.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Has anyone seen any use cases for additional "usages" in the wild while this MSC has been implemented? @deepbluev7 perhaps?
If not, perhaps it's just best to leave this as is.
@@ -0,0 +1,269 @@ | |||
# MSC2545: Image Packs (Emoticons & Stickers) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Stickers support in kazv: https://lily-is.land/kazv/kazv/-/merge_requests/71 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hello all. I plan to breathe new life into this MSC by addressing any outstanding concerns and making changes as necessary.
I'm aware that this MSC has already been successfully implemented in numerous clients (thank you!), and I intend to remain compatible with those implementations as much as possible.
I very much look forward to seeing custom emotes and stickers become a part of the Matrix spec. Let's make it happen!
proposals/2545-emotes.md
Outdated
For emoticons a client could add deliminators (e.g. `:`) around the image shortcode, meaning | ||
that if an image has a shortcode of `emote`, the user can enter `:emote:` to send it. If there are |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a shortcode grammar defined anywhere? I see a definition of "must be a string starting and ending with a :
character" and "must be less than 100 bytes" in MSC4027. But that seems like an odd MSC to be the one to officially define the grammar for shortcodes.
Perhaps this MSC could adopt the same rules, and MSC4027 reference it instead?
set one state event *per image*. While this might seem like a nice idea on the surface, it doesn't | ||
scale well. There are people who easily use and want hundreds or even thousands of image packs accessible. | ||
A simple dict of shortcode to mxc URI seems more appropriate for this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a go at calculating a rough estimate of how many images (as defined by this MSC) you could fit into a single 65535 byte state event: https://github.com/anoadragon453/emote-estimate
The result was ~400 emotes per pack. This actually seems fairly tolerable for every day usage, especially as Discord has a maximum of 250 emotes per server (assuming you pay for that limit bump).
The main concern I can see is bridging; Slack seemingly lets you have infinite custom emoji per workspace, so a bridge would need to split that up into multiple packs, which may look a little ugly in the UI.
But my opinion is that 400 emotes is fine for now, and people are getting along with the current limits. A future MSC could add a method for merging image packs defined in separate state events together, but I don't think the current limit is a blocker.
image pack event is in the room. This does not have to be unique from other packs in a room. | ||
- `avatar_url`: (String, optional) The mxc uri of an avatar/icon to display for the pack. Defaults | ||
to the room avatar, if the pack is in the room. Otherwise, the pack does not have an avatar. | ||
- `usage`: (String[], optional) An array of the usages for this pack. Possible usages are `"emoticon"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I feel the usual spec process of being able to add unstable prefixes to values in this field is sufficient? The current wording doesn't prevent that.
3. Space image packs (defined in the hierarchy of canonical spaces for the current room) | ||
4. Room image packs (defined in the currently open room's state) | ||
|
||
Furthermore, clients are expected to de-duplicate images so that same images are not displayed multiple |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It also makes it more difficult to manage packs, in case you want to manually remove duplicates.
Perhaps it would make more sense to only de-duplicate packs themselves?
Whenever they are defined.
Any updates on this? |
Rendered
Signed-off-by: Sorunome [email protected]
Author: @Sorunome
Future MSCs
Current implementations
Emote rendering (rendering of the
<img>
tag)data-mx-emoticon
)data-mx-emoticon
)Sending, using the mentioned events here
Bridges
data-mx-emoticon
only)Implementation PRs
FluffyChat
Data model: https://gitlab.com/famedly/company/frontend/libraries/matrix_api_lite/-/merge_requests/26
SDK: https://gitlab.com/famedly/company/frontend/famedlysdk/-/merge_requests/726
Emoticons: https://gitlab.com/famedly/fluffychat/-/merge_requests/433
Stickers: https://gitlab.com/famedly/fluffychat/-/merge_requests/452
Nheko
Stickers: Nheko-Reborn/nheko#648
Sticker editor: Nheko-Reborn/nheko#669
Choosing emoticons: Nheko-Reborn/nheko@ea6b19b
Cinny
Emoticons and Stickers: cinnyapp/cinny#686
kazv
Creating and sending stickers: https://lily-is.land/kazv/kazv/-/merge_requests/71