diff --git a/proposals/2448-blurhash-for-media.md b/proposals/2448-blurhash-for-media.md new file mode 100644 index 00000000000..cc321b906ea --- /dev/null +++ b/proposals/2448-blurhash-for-media.md @@ -0,0 +1,377 @@ +# MSC2448: Using BlurHash as a Placeholder for Matrix Media + +[BlurHash](https://blurha.sh) is a compact representation of a placeholder +for an image (or a frame of video). Currently, in Matrix, clients must +display a placeholder image in the message timeline while a piece of media is +loading. Some clients, such as Element, simply display an empty space. + +While thumbnails exist to combat this to some degree, they still need to be +downloaded from a homeserver, which is not instantaneous. + +Instead, a BlurHash can be sent inside events, which upon +receipt other clients can render for a pretty media preview while the actual +thumbnail downloads. They also do not contain any `"` characters, making them +simple to stick inside existing JSON blobs. + +To be clear: A BlurHash does not replace a thumbnail - rather it is a placeholder +and should be shown before the thumbnail is downloaded. For a familiar messaging +user experience, clients are recommended to first render and display a blurhash, +then download the thumbnail of the media. Once the thumbnail file is downloaded, +display it. Finally, the user can optionally view the full media item by selecting it. + +## Proposal + +### m.room.message + +An optional field is added in `m.room.message`'s `content.info` dictionary +with the key `blurhash`. It is a BlurHash of the original piece of media. +Clients could then render this using [one of the available BlurHash +implementations](https://github.com/woltapp/blurhash). + +This would be optionally displayed while the thumbnail of the media is loaded +in parallel. + +Example `m.room.message` content: + +```json +{ + "body": "image.png", + "info": { + "size": 149234, + "mimetype": "image/png", + "thumbnail_info": { + "w": 301, + "h": 193, + "mimetype": "image/png", + "size": 72958 + }, + "w": 301, + "h": 193, + "thumbnail_url": "mxc://example.org/abcdefg", + "blurhash": "JadR*.7kCMdnj" + }, + "msgtype": "m.image", + "url": "mxc://example.org/abcde" +} +``` + +Note that a BlurHash representation is really only applicable to media, and +as such should only be used in conjunction with the following +`m.room.message` msgtypes: + +* `m.image` +* `m.video` + +### m.sticker + +An optional field is added to `m.sticker`'s `content.info` dictionary with +the key `blurhash`. Its value is a BlurHash of the sticker media. + +Example `m.sticker` content: + +```json +{ + "body": "Landing", + "info": { + "h": 200, + "mimetype": "image/png", + "size": 73602, + "thumbnail_info": { + "h": 200, + "mimetype": "image/png", + "size": 73602, + "w": 140 + }, + "thumbnail_url": "mxc://matrix.org/sHhqkFCvSkFwtmvtETOtKnLP", + "w": 140, + "blurhash": "JadR*.7kCMdnj" + }, + "url": "mxc://matrix.org/sHhqkFCvSkFwtmvtETOtKnLP" +} +``` + +### m.room.avatar + +Room avatars having BlurHashes available will be especially useful when +viewing a server's Public Rooms directory. + +An optional field is added to `m.room.avatar`'s `content.info` dictionary with the +key `blurhash`. Its value is a BlurHash of the media that is pointed to by +`url`. + +Example `m.room.avatar` content: + +```json +{ + "url": "mxc://matrix.example.com/a59ee02f180677d83d1b57d366127f8e1afdd4ed", + "info": { + "blurhash": "JadR*.7kCMdnj" + } +} +``` + +### m.room.member + +Much like room avatars, user avatars can have BlurHashes as well. There is a +little more required to implement this, but the outcome of no longer having +missing avatars upon opening a room is worthwhile. + +An optional field is added to `m.room.member`'s `content` dictionary with +the key `blurhash`. Its value is a BlurHash of the media that is pointed +to by `avatar_url`. + +Note that `blurhash` MUST be omitted if `avatar_url` is not present. + +Example `m.room.member` event content: + +```json +{ + "avatar_url": "mxc://example.org/SEsfnsuifSDFSSEF", + "displayname": "Alice Margatroid", + "membership": "join", + "blurhash": "JadR*.7kCMdnj" +} +``` + +### Profile endpoints + +Endpoints that return profile information, and thus MXC URLs to user avatars, are +extended to optionally include BlurHashes as well. + +[`GET /_matrix/client/v3/profile/{userId}`](https://spec.matrix.org/v1.3/client-server-api/#get_matrixclientv3profileuserid) +has an optional field added with +the key `blurhash`. Its value is a BlurHash of the media that is pointed to +by `avatar_url`. `blurhash` MUST be omitted if `avatar_url` is not present. +The same applies +to [`GET /_matrix/client/v3/profile/{userId}/avatar_url`](https://spec.matrix.org/v1.3/client-server-api/#get_matrixclientv3profileuseridavatar_url) +, and to the federation +endpoint [`GET /_matrix/federation/v1/query/profile`](https://spec.matrix.org/v1.3/server-server-api/#get_matrixfederationv1queryprofile) +. + +#### Example responses: + +`GET /_matrix/client/v3/profile/{userId}` + +```json5 +{ + "avatar_url": "mxc://matrix.org/SDGdghriugerRg", + "displayname": "Alice Margatroid", + "blurhash": "oyp8ky2BWn7VHEL" +} +``` + +`GET /_matrix/federation/v1/query/profile` + +```json5 +{ + "avatar_url": "mxc://matrix.org/SDGdghriugerRg", + "displayname": "Alice Margatroid", + "blurhash": "oyp8ky2BWn7VHEL" +} +``` + +`GET /_matrix/client/v3/profile/{userId}/avatar_url` + +```json5 +{ + "avatar_url": "mxc://matrix.org/SDGdghriugerRg", + "blurhash": "oyp8ky2BWn7VHEL" +} +``` + +Separately, [`PUT /_matrix/client/v3/profile/{userId}/avatar_url`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3profileuseridavatar_url) +has an optional field added +to the request body with the key `blurhash`. Its value is a BlurHash of the media that is pointed to by the value of +the `avatar_url` field in the same request. + +#### Example request/response interaction + +Request: +``` +PUT /_matrix/client/v3/profile/{userId}/avatar_url HTTP/1.1 + +{ + "avatar_url": "mxc://matrix.org/SDGdghriugerRg", + "blurhash": "oyp8ky2BWn7VHEL" +} +``` + +Successful response: + +``` +200 OK + +{} +``` + +### URL previews + +An optional attribute is added to the OpenGraph data returned by a call +to +[`GET /_matrix/media/v3/preview_url`](https://spec.matrix.org/v1.3/client-server-api/#get_matrixmediav3preview_url) +called `matrix:image:blurhash`. The value +of this attribute is the blurhash representation of the media specified +by `og:image`. Note that we place this value under the `matrix` namespace as +the [OpenGraph protocol](https://ogp.me/) does not (yet) have a spec'd field +for BlurHashes. + +Note that `matrix:image:blurhash` MUST be omitted if `og:image` is not present +in the response. + +Example response to `GET /_matrix/media/v3/preview_url`: + +```json +{ + "og:title": "Matrix Blog Post", + "og:description": "This is a really cool blog post from matrix.org", + "og:image": "mxc://example.com/ascERGshawAWawugaAcauga", + "og:image:type": "image/png", + "og:image:height": 48, + "og:image:width": 48, + "matrix:image:size": 102400, + "matrix:image:blurhash": "oyp8ky2BWn7VHEL" +} +``` + +### Inline images + +An optional attribute is added to `` tags in messages: +`data-mx-blurhash`, where the value of the attribute is the blurhash +representation of the inline image. + +This would be optionally displayed while the inline image itself is loaded in +parallel. + +Example `m.room.message.formatted_body`: + +``` +"formatted_body": This is awesome \"flutterjoy\" +``` + +## Calculating a blurhash on the server + +BlurHashes are inserted into events by the client, however some clients may not +be able to implement the BlurHash library for whatever reason. In this case, it +would be nice to allow the media repository to calculate the BlurHash of a piece +of media for the client, similar to how thumbnails are calculated by media +repositories today. + +A new boolean query parameter, `generate_blurhash`, is added to +[`POST /_matrix/media/v3/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixmediav3upload) +which allows clients to ask the server to generate a BlurHash for them. This +may be useful for clients that are designed to run on very low-power hardware, +or otherwise cannot generate a BlurHash itself. +If set to `true`, the server SHOULD generate a BlurHash of the uploaded media +and return it as the value of the `blurhash` key in the response. + +If the server cannot generate a BlurHash of the media - perhaps because it is +not a file that a BlurHash can be derived from or it is too expensive to process +a very large piece of media - then the server SHOULD NOT return a `blurhash` key +in the response. Additionally, if `generate_blurhash` is not `true`, then the +server SHOULD NOT return a `blurhash` key in the response. + +Fundamentally, this means that clients SHOULD NOT assume that a server will always +return a BlurHash in the response to `/_matrix/media/v3/upload`, even if they have +set the `generate_blurhash` query parameter to `true` in the request. + +An example request from a client that would like the server to generate a +blurhash would look like: + +``` +POST /_matrix/media/v3/upload?generate_blurhash=true&filename=My+Family+Photo.jpeg HTTP/1.1 +Content-Type: image/jpeg + + +``` + +Example response: + +``` +{ + "content_uri": "mxc://example.com/abcde123", + "blurhash": "LKO2?U%2Tw=w]~RB" +} +``` + +We explicitly make this behaviour opt-in as it is assumed that the majority of +clients that end up supporting BlurHashes will be capable of generating them +locally. Thus, the less load we can put on the homeserver (by not making +blurhash generation the default) the better. + +Note that media servers will not be able to return a BlurHash string for +encrypted media; that must be left to the client. + +The server could additionally return the BlurHash string for an image when +given an MXC URL. This would be through something like the Media Information +API (specified in +[MSC2380](https://github.com/matrix-org/matrix-doc/pull/2380)), or similar. + +## Visualisation + +Viewing an image message that is loading: + +![blurhashed preview](images/2448-blurhash.png) + +Once the image loads: + +![the image has loaded](images/2448-loaded-image.png) + +As a sample, this is Element's behaviour prior to this MSC's introduction: + +![boo, sad](images/2448-current-state.png) + +## Alternatives + +We could include a base64 thumbnail of the image in the event, but blurhash +produces much more efficient textual representations. + +## Backwards compatibility + +Older clients would ignore the new `blurhash` parameter. + +Newer clients would only show it if it exists. + +Users who have not specified `blurhash` in their `m.room.member` event yet may +stand out from users who have while both are loading. This is entirely up to +clients to handle, though a suggestion may be to "fake" a blurhash by +blurring some placeholder image (perhaps something with variation between +users like Element's coloured backgrounds with letters in them, or [an +identicon](https://en.wikipedia.org/wiki/Identicon) derived from the user's +ID) until the user's actual avatar loads. + +## Unstable prefixes + +Implementations wishing to add this before this MSC is merged can do so with +the following: + +* The `blurhash` key in any events, request or response bodies should be + replaced with `xyz.amorgan.blurhash`. + +* `/_matrix/media/v3/upload`'s new `generate_blurhash` query parameter + should instead be `xyz.amorgan.generate_blurhash`. And instead of the + key `blurhash`, the endpoint should return `xyz.amorgan.blurhash`. + +* The `data-mx-blurhash` attribute in `` tags should be replaced with + `data-xyz-amorgan-blurhash`. + +## Security considerations + +BlurHash entries in encrypted events, be it as part of the `info` property, +or `` tags, should be encrypted along with the rest of the event +content. + +A [discussion in +#matrix-spec](https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$Cfa0dtF3DenIUAbC5aeg3Xo10gAF54mAJLZ6VzvYNfo?via=matrix.org&via=amorgan.xyz&via=pixie.town) +questioned whether massive BlurHashes may be a potential DoS vector for +clients. The discussion found that only a maximum of 100 x 100 components can +be defined by a BlurHash. This may be in the higher range for low-resource +(or unoptimised) clients, and clients are free to refuse to render a BlurHash +with a large component count, but it shouldn't be a cause for concern. + +Invalid BlurHashes should not be rendered. + +## Links + +BlurHash's algorithm description can be found +[here](https://github.com/woltapp/blurhash/blob/master/Algorithm.md), which +also includes the full output character set. diff --git a/proposals/images/2448-blurhash.png b/proposals/images/2448-blurhash.png new file mode 100644 index 00000000000..5136884fcb4 Binary files /dev/null and b/proposals/images/2448-blurhash.png differ diff --git a/proposals/images/2448-current-state.png b/proposals/images/2448-current-state.png new file mode 100644 index 00000000000..bf02d57a8c3 Binary files /dev/null and b/proposals/images/2448-current-state.png differ diff --git a/proposals/images/2448-loaded-image.png b/proposals/images/2448-loaded-image.png new file mode 100644 index 00000000000..fc4656364fe Binary files /dev/null and b/proposals/images/2448-loaded-image.png differ