MSC3552: Extensible Events - Images and Stickers

[turt2live]

Rendered

Blocked by #1767
Blocked by #3551

Preview: https://pr3552--matrix-org-previews.netlify.app

[HarHarLinks]

I want to bring to your attention that imo Stickers should also allow video/mp4, video/webm etc as gif replacements. My point is, I'm not sure if Images and Stickers should be lumped together.

[turt2live]

@HarHarLinks please use threads on the diff for feedback to be considered.

Accepting videos would be an entirely different MSC, I think.

[tulir]

MSC1767 already specifies "m.emote": {} and "m.notice": {} in the event content for extensibility purposes:

In the case an event needs to fallback to m.emote or m.notice, the appropriate type can be included in the event content

Shouldn't that apply here too? Otherwise there's no way to tell clients that a custom event should fall back to a sticker.

[turt2live]

@maltee1 says:

What about sending multiple images in a single message with a single caption? I imagine people would want to send an "album" rather than cluttering the timeline with individual messages, if they have lots of related pictures to share. It's also something that other platforms support (I know signal does) and would improve bridging if it's available in matrix.

[turt2live]

Events in Matrix are intended to represent a single unit of information - it's already a bit questionable to have a caption on an image in the same event, but there are slightly more pros than cons for doing so (currently). Albums would instead be represented by a series of image events, linked together using relationships of some kind, potentially with an "information event" to store things like the caption.

A relationship-based system would allow for richer support too: adding images/videos after the fact, mixed content, content from other senders, edits, better organization (moving images between albums), etc. Representing the whole thing as one event gets complicated to manage at a technical level.

[maltee1]

Excuse my ignorance, but wouldn't splitting up an album in several events lead to the same problems that splitting up caption+imagine into two events would have? For example a bridge not knowing how long to wait for related events to appear before bridging.

[turt2live]

Some of the problems are shared, though from looking at it in the past it always felt like albums should be distinct events.

Regardless, albums would be handled by another MSC (this MSC defines an image format and is tightly scoped to that).

[TheArcaneBrony]

For what it's worth, I see value in adding caption+image in the same event from an accessibility standpoint (see: alt text in HTML/some social media platforms)

[turt2live]

@maltee1 says:

Another question: Does it make sense to offer an image in several resolutions rather than distinguishing between a thumbnail and the full image? I know of at least one platform (WeChat) that offers sending and receiving images in two resolutions, one that should be sufficient for most phone screens and "full size", which is just the original image. I don't know exactly how the protocol works, but I suppose it also uses thumbnails, which brings the number of resolutions to 3. The current proposal allows for several thumbnails, which could be used to represent that, but defining an image as a thumbnail implies that it shouldn't be used to show full-screen, even if the resolution may be sufficient. Instead, we should possibly not distinguish a "thumbnail" but simply offer several resolutions as desired (with the suggestion that one of them is thumbnail-sized) and have clients pick the one they want to show for a given purpose. Depending on available image resolution and expected bandwidth a thumbnail might even be unnecessary but we want to include a high-res version. The current proposal is not designed for that.

[turt2live]

In short, a 1080p thumbnail wouldn't be unreasonable to include here. Some clients already rely on "high quality" thumbnails, which would be represented here.

Clients are already asked to find the thumbnail that works for them.

---

As an aside, please use comments on the diff - comments not on the diff are likely to get ignored.

[maltee1]

Thanks for your patience, I will reply to the diff from now on.
If clients are expected to go through all available image sizes and pick the one that suits a particular purpose, isn't it just complicating implementations to include all sizes but one in a single list and keep the largest size separate? What's the meaning of "thumbnail" in that case?

[turt2live]

It's mostly to deliminate "original copy" and "machine-resized" images.

[cloudrac3r]

In my view, creating a different event type is less "extensible". With this proposal, it is possible to add a text caption to an existing m.image event by replacing (editing) it and adding the caption, but it is not possible to add an image to an existing m.message text event, since you cannot replace the type of an event. This doesn't make any sense. Text messages and image messages should both be candidates for being edited into a text+image message.

I think it would make more sense if the m.message event was more extensible and allowed the inclusion of text, images, or both.

Is there any rationale for making it a different message type?