Render images directly from the terminal

[neverping]

Back in time, terminals were always text-based.

Currently, almost everyone uses a terminal on a GUI environment. It amazes me to not having a terminal that doesn't render a image if I do a "cat sunset.jpg".

It would be nice if Rio could detect such output and renders it. Or maybe offering a built-in command to display them, maybe "draw image.jpg".

[raphamorim]

That would be a nice to have for after the first release. Marking it as wish list.

Believe kitty terminal also does that with icat: https://sw.kovidgoyal.net/kitty/kittens/icat/

[chase]

The basic protocol for images in Kitty is pretty simple:
https://sw.kovidgoyal.net/kitty/graphics-protocol/

I used it to make a simple but fully functional Chromium-based web browser that renders in Kitty and WezTerm (with Kitty protocols enabled): https://github.com/chase/awrit

There is also the iTerm2 image protocol which has support in a lot of newer terminal emulators: https://iterm2.com/documentation-images.html

[AnonymouX47]

The basic protocol for images in Kitty is pretty simple:

Point of correction... it's not that "simple" on the end of the terminal emulator.

It's probably the main reason why no terminal emulator other than Kitty implements nearly half of the protocol [correctly].

[chase]

The basic protocol for images in Kitty is pretty simple:

Point of correction... it's not that "simple" on the end of the terminal emulator.

It's probably the main reason why no terminal emulator other than Kitty implements nearly half of the protocol (correctly).

Fair enough, certainly the protocol in its entirety is complicated.

The scope of this issue isn't to support image IDs, arbitrary positions, animation etc. It's to render images directly in the terminal, and that can be done with a subset of the protocol, described within the first few paragraphs.

A minimal subset, similar to iTerm2's protocol, can be implemented: permitting only PNG filenames to be transmitted and displayed immediately at the current cursor position. Then handling a clear screen to remove said image.

Although you certainly have more experience with such things than myself, I wouldn't want the maintainer to get discouraged. Incremental improvement and all that.

[AnonymouX47]

The scope of this issue isn't to support image IDs, arbitrary positions, animation etc. It's to render images directly in the terminal, and that can be done with a subset of the protocol, described within the first few paragraphs.

Defining that "subset" is where the problem begins.

A minimal subset, similar to iTerm2's protocol, can be implemented: permitting only PNG filenames to be transmitted and displayed immediately at the current cursor position. Then handling a clear screen to remove said image.

Although you certainly have more experience with such things than myself, I wouldn't want the maintainer to get discouraged. Incremental improvement and all that.

From the little experience I have with the two protocols mentioned here, I'll advice the iTerm2 protocol be implemented first (because it's a lot less complex than Kitty's) and the Kitty graphics protocol should only be implemented when the project is more mature and developers/contributors are ready to take it on properly.

For some insight, check out the current state of support in Wezterm (just a search through the issues on there will do). By the way, the support is declared experimental but comparing it to the state of the iTerm2 protocol support (and how long each has been in development) makes my point clear.

Please Note: The above mention/reference is not intended to talk lightly/negatively about the project or its author/developers (in fact, Wezterm is my daily driver and it's doing very well in almost every other aspect I can think of).

The summary of all I've said is... It's better to have a [almost] complete, spec-compliant implementation of a less-capable protocol than a highly incomplete implementation of a more-capable protocol. Also, it'll be less headache for the developers.

---

That said, there's also sixels (requested in #46) but that has its own unique set of headaches, primarily due to the lack of a complete central/standard specification and limited access to a reference implementation (in fact, the closest to a reference implementation I've come across is probably not something any relevant implementation today would want to follow head-on).

Anyways, sixels are probably still simpler to implement than the kitty graphics protocol given the limited scope.

I should note that my experience with sixels is currently limited, there are others way more knowledgeable/experienced on the topic.

[AnonymouX47]

Hello @raphamorim !

I noticed some sixel implementation work was underway, though, I'm not sure of the state as I'm yet to test.

I just wanted to inform you about an ongoing related discussion that's quite important as I believe it's important that things are done well and correctly right from the onset.

See hpjansson/chafa#192, you might wanna follow along and possibly chime in.

Well done and thank you.

[raphamorim]

Next version will include an initial version of sixel and iterm2 protocol. Regarding kitty image protocol I will work in the future.

[AnonymouX47]

I just tested it out the sixel support...

First, kudos for the work. 👍

Just one observation so far... text doesn't overwrite sixels, which I believe is opposite to the standard behaviour, as in:

This is unlike several other terminal emulators. Also, this means the only way to clear sixels off the screen is by actually clearing the whole screen which isn't very practical. This might not seem like a problem to users who only "cat out" images on a scrolling screen but this behaviour makes the feature difficult/impossible to use in fullscreen applications (i.e. which don't scroll the screen). Below are shots from Xterm and WezTerm:

Xterm	Wezterm

For what it's worth, I understand the work isn't finalized yet but I thought it was still worth noting.

Finally, I see iTerm2 IIP support is also underway. I'd like to note that the behaviour should be the same.

Thanks. 😃

[AnonymouX47]

On a different note, I noticed parameters 14 and 16 of CSI t aren't implemented [yet]. Though, according to the docs, they aren't rejected. Some applications require at least one of these two (usually 14) to use graphics protocols.

For what it's worth, I know the TIOCGWINSZ ioctl is implemented already (and that's personally what I always prefer) but it isn't an option in every environment e.g on Windows.

[raphamorim]

@AnonymouX47 thank you so much for your notes regarding sixel implementation. Could you create an issue regarding that? I will take a look on that before cut the release 🙏

also regarding 14 and 16 if could be another issue would be awesome. It helps me to keep track of the stuff to do.

and yes definitely looking for help on iterm2 validation/testing

[orhun]

I also tried this out from the main branch using ratatui-image's demo (cargo run --example demo --features crossterm):

Meanwhile in wezterm:

How can I debug this? 🤔

[raphamorim]

@orhun please open a new issue with it?

I will lock this issue for now because sixel issues should be documented in different issues (easier for me to track lol)