Draft for playon compatibility with the jellyfin server

[pinsarda]

This is a very early shot at getting the playon feature from jellyfin to work with finamp.
For now this is a extremely rough draft to start working on the feature and getting rid of an annoying bug. Can only start songs without having media controls and need to enter the url manually.

[Chaphasilor]

Thanks for getting this started, really looking forward to it!

So I managed to get this working a bit more reliably and generically some time ago, just pushed those changes :)
The playback controls are now all shown, and Finamp registers support for all commands I could find. The important ones are already implemented too.
I didn't work on the actual "play on" functionality though, so that's unchanged, meaning it works for the most part, but sometimes the wrong item is picked (e.g. selecting a specific track from an album will play the first track from the album instead). Ideally, this should work just like it works in Finamp, and make use of the same settings. So, if a single track is selected, depending on the Finamp settings it should either play just that track or start an instant mix from that track.

There are still issues with the websocket, it seems to get disconnected (or at least stops receiving messages) after a short while or certain commands. Haven't looked into that just yet.
I also noticed when testing this out again that when logging into the server the websocket connection will fail until the app is restarted, because the connection is established before the URL and credentials are know. The initialization should therefore be delayed until setup is complete, or restarted after successfully logging into a server.

[pinsarda]

I apologise for taking so much time to go back to this, but as they say better late than never ! I didn't find the source of the issues with the websocket, still investigating. For now I quickly fixed the item selection, now it starts the instant mix and the correct song as it should.

[pinsarda]

The current implementation works very well on my phone, but can probably be a bit slow as I request each song individually instead of the whole album at once. I don't know if there's an easy way to do it better currently...

[Chaphasilor]

Well just take a look at how the content of an album is fetched for the album screen :)
It's just a matter of setting the parentID filter, I think.

[pinsarda]

Yes I thought about that but this would introduce a bug when starting a playlist. The websocket just gives us ids of songs to play. It doesn't ask finamp to play an album or a playlist. This makes it hard to do it differently.

[Chaphasilor]

Is there no way at all to differentiate from "where" the track was selected? I guess in that case all we can do is play just that one track (or start the instant mix, depending on settings), and nothing else.
Is that also the case when you try to play an entire album or playlist? Does it only send us the first track, without any additional information?

[pinsarda]

I don't think so, we just receive a list of ids, and optionally the index of the track selected if there was any. The current implementation does what you're proposing. If a specific track is selected, we only play this one or start instant mix. And if there's no precision on the starting index it queues the whole album/playlist. After testing it seems like the best possible behaviour to me.

[pinsarda]

As discussed in #500 an internal volume controller will be implemented eventually, we'll rely on that instead of system wide volume.

[Chaphasilor]

@Maxr1998 how did you approach this issue (playing albums/playlists) on the Android (TV) app? 🤔

[Maxr1998]

Can't speak for Android TV, not even sure play on is supported there. On the Android app, we just wrap the web app so the handling is identical to that.

[Komodo5197]

This seems like the server is giving us a list of song IDs to form the queue, which means the most similar piece of code is loadSavedQueue() in lib/services/queue_service.dart, so you can probably look at that for inspiration. Most notably, you can make batch requests for up to 200 item IDs at once, which is significantly faster. Also, I'm pretty sure the behavior of loading the single song or an instant mix is questionable. We should be loading the whole id list from the server into the queue and just setting the initialIndex to what the server tells us to.

[Chaphasilor]

@Komodo5197 the playing of a single track should only happen when the user selects a single track in the other client, and the API only gives us one ID. If we get multiple IDs, we should play all of them of course.

Thanks for the suggestion about the queue restore, that would make sense.

However, my main concern was how to identify where these tracks are coming from, so we can show a proper queue source. If the user requests to play an album, I'd like Finamp to show which album is playing as the queue source name (so that one could also tap the source at the top of the player screen, and immediately go there). Same for playlists, artists, etc. But it seems that info is simply not given to us, so we'll have to default to some generic ("Queued by Remote Device"?) name instead.

[Komodo5197]

We are getting the full queue. I click a track partway through an album on the web UI and finamp recieves {"MessageType":"Play","MessageId":"560c155a593a497e8bd047610f2e8e4a","Data":{"ItemIds":["46430779c8072941ed8f7e2ddafaebe4","f8ee032bb75db53c1c317c572e3e8814","fda9dba1e21b261bdb050efe5bd95bbb","4b0a4b5c756079eb8322860215fbe5fb","e012de5b9fb821a2258b6e91c9225459","ea8e6422f6266c453469e104e3e4b950","cbb922f4f858571b79dafa8da3578270","d891ee5acb57e58ca338df5518c8ce37","80ed7b24b4878c28a75e6234ff066bd6","ca4e6f99c7ad2bae487aaba0585f869c","d586e2361bab084b04872a4128507db8","67f19176d6582739abfd0678442b45de","c34b8e312fd99b7194d927c64a9700a9","c89368641479249da0b333c09ba1aa40","ba8ade9d002d8c40bae852531bcebba5","8ac7aa4480fa7fcaecf1d03a816431ed","22a0587b3018e44644924535542ace41","ca63bf02ba1a0bee484e2fb6e036d3e0"],"PlayCommand":"PlayNow","ControllingUserId":"de6a683bafb44247881bb62bf125f7e8","StartIndex":8}}.

There are also additional play commands 'PlayNext', corresponding to addToNextUp(), and 'PlayLast', corresponding to addToQueue(). Implementing those allows remote queue building that seems to match up with how the web client works.

Regarding queue source, it does indeed seem that we don't have this information in any form, and will need to use a generic header.

Also, I happen to still be running a 10.8 server, and to get this to work I had modify the ClientCapabilities to explicitly set supportsSync and supportsContentUploading to false, and remove the supported command "SetPlaybackOrder".

[Chaphasilor]

Okay, thanks for taking a look. Haven't had a look at the code recently, but there are still instances where a single track is selected, e.g. when playing from a search result. In that case, the Instant Mix behavior should still apply, I think.

Interesting note about the 10.8 server. I also still have one of those lying around, I'll give it a try myself. Maybe the commands had a different name back then? I'd like to keep the playback order functionality if possible, but I'd also like to keep supporting 10.8.
Maybe it's finally time to query the server info (version, name, etc.) on startup, and have some version-specific features. I guess feature detection is a hopeless endeavour with Jellyfin anyway...

[Chaphasilor]

I gave the current version a try yesterday. It was working well at first, and I could see the queue, playback info, and control playback mostly fine. But later on I didn't manage to get the queue or even the playback info to show up anymore, even after restarting the app.
I also encountered an issue where it seems like commands where being queued for some reason. I tried playing/pausing and sending a few other commands (including a message) through the web client to Finamp, which didn't have an effect. But then many minutes later I tried to pause playback (in Finamp directly), but it was resumed immediately. That happened a few more times, and then finally it actually did pause and stayed paused, and then the message appeared. Maybe there's a command waiting for a Future that never resumes, or something similar?

[pinsarda]

For the first issue, it's weird I didn't experience it. I got the same bug regarding command queuing though, and got to the same conclusions as you without nailing it yet. I'll get back to it tomorrow and on thursday. I'll also adjust the code to have the behaviour proposed by @Komodo5197

[Chaphasilor]

Just wanted to note that the websocket connection should get disconnected when switching to offline mode, before I forget about it.

[pinsarda]

There's an issue when using playlist, the songs are completely unordered. This is an upstream bug from jellyfin : jellyfin/jellyfin#8885
I didn't look into the details just yet

[Chaphasilor]

That's because the server sends us the raw track list instead of just the playlist ID, right? If there is an ID / some way to figure out which playlist is being requested, we could simply disregard the track list and fetch the playlist ourself?

[pinsarda]

Absolutely, but the playlist id is not in the message received by the websocket unfortunately. Plus you have no way to even know if it's a regular album or a playlist being played (apart from checking if all the songs belong to the same album after fetching data). As you guessed, we only get a list of ids of songs to play and a starting index, that's it.
I have dirty workarounds in mind but I would rather have the bug fixed properly jellyfin side. And/or have the playlist/album id supplied directly in the websocket data, which would also be nice to populate the "playing from" label.

[pinsarda]

In the last commit, I am not too sure about my "updateState" function in favoriteProvider, I couldn't find an other way to do it directly using riverpod's ref. It works but may be unnecessary to create this function.

[Chaphasilor]

I think we should have a getter/function in playon_handler.dart that generates these ClientCapabilities. So that the file that handles the commands also decides which commands are currently supported.

We can also use the proper type for "Audio", BaseItemDtoType.song. I'm also wondering if some other types should be supported, like MusicAlbum, MusicArtist, MusicGenre, Playlist, and AudioBook. Maybe if those are supported the server will actually send us some information about the playlist or album being played, instead of just a list of tracks?
Scratch that, it's a different enum: https://api.jellyfin.org/#tag/Session/operation/PostCapabilities

[Chaphasilor]

we should probably check somewhere if we're already connected, otherwise we'll reconnect every time a setting is changed! And since volume is a setting, that might be causing issues...