What happened to SFH compatibility?

[Whiskers11]

When ever I try changing my NONG it says "Failed to set song: file not found," and when I try finding new NONG's it just brings me to https://songfilehub.com/home. What happened?

[Fleeym]

song file hub support was dropped in the last update, you can check this release out for more info
https://github.com/Fleeym/jukebox/releases/tag/v2.2.0

[PmikeGamer]

the whole point i got this mod is GONE now
so annoying

[Fleeym]

it is what it is

[whitemaskedwolf]

soo did you just straight up erase the entire git tree up to the update which removed song file hub integration?

people downloading all of the applicable songs from the song file hub manually is ultimately going to have the same impact on the servers as the mod doing it automatically, so i don't even see the reason to remove the functionality in the first place, let alone undermine version control to prevent people from gaining access to previous versions of the mod.

most people don't have the knowledge required to build the mod for themselves, and an even smaller subset of those people are going to be bothered to actually do so. unless you hid instructions for building a thermonuclear bomb in the previous revisions or something, it wouldn't have been dangerous to simply not erase the entire repository.

but welp, i can't change the past anyhow, so if you think this was the appropriate thing to do then you do you.

[Fleeym]

the difference here is convenience. it was much more convenient to just click set on different songs you had fetched compared to downloading each one in particular and checking it out. also the website has ads, those balance out expenses a little
i myself am not sure if that sudden jump of usage was because of jukebox, but it surely did align with its increase in popularity, especially among android players. for now, i think i did the right thing. if we figure out a solution, i would be glad to bring back the integration :)

[RoootTheFox]

hi @Fleeym, sorry for "hijacking" this issue. I would be willing to host a mirror of all songs on SFH, including an API for mods like Jukebox.

I would also be willing to work together with the SFH team to make the mirror/API as seamless as possible.

My current plan/idea consists of a small web service, that:

• periodically monitors SFH for changes
  • (every hour or so, maybe even through a webhook or similar to make updating songs faster)
• saves the songs
• and provides an API for integrations like Jukebox.

I already have a server (VPS), so hosting wouldn't be an issue for me.

That way this mod would no longer stress out SFH's servers, while being able to retain the convenient in-game NONG downloading function that everyone misses.

Is this something you'd be willing to implement?

[Fleeym]

Sure, the main problem that SFH encountered was outbound traffic limits. Jukebox took about 600GB / 4 days, and their monthly limit was about 6TB. Obviously, Jukebox's consumption rate would only increase as time goes on. Do keep this in mind when going through with your idea.

[RoootTheFox]

Okay, that is a lot of data (and now I fully understand why you dropped SFH compatibility)
My server has an outgoing traffic limit of 32TB(/month?) so I don't think traffic limits are an issue (I don't think bandwidth will be problematic either; if my calculations are correct it should average out to about 2MB/s, although this will obviously be way higher during peak hours)

I have two other ideas:

• (relatively no-brainer to me): CloudFlare CDN/proxying the actual song files, I'm pretty sure even the CF free tier could make a pretty huge difference here
• a different server architecture that works like this:
  • Jukebox -> main API/load distributor (selects a mirror based on server load or RNG, returns URL for the song on that mirror) -> Jukebox (requests the song from that specific mirror)
  • That way, hosting of songs can be more decentralized, reduce load on individual mirrors and mitigate downtime if a mirror goes offline for a bit
  • (There could obviously be multiple load balancers run by different people too, that way load balancer downtime could be mitigated as well)
  • That way, people with the necessary server infrastructure that want to help out can volunteer to host both mirrors (where the actual song files are saved) and optionally load distributors for higher reliability, allowing easy scaling and therefore making everything more future-proof.

[Fleeym]

I think just CloudFlare should improve things by quite a lot. I'm sure it can also cache stuff a little more aggressively, since the files don't really change. That should be a good enough start.

[RoootTheFox]

@Fleeym i made an initial version of the mirror API server, source code can be found here (I'll do proper error handling tomorrow, as well as setup documentation)

The API (and mirror) are currently up at https://sfh.rooot.gay

the JSON API is pretty self-explanatory, currently there are 3 endpoints:

• /api/v1/get_song/<id> - id is the SFH internal id (_id); returns a single JSON object of the song
• /api/v1/get_songs_for_level/<level_id> - <level_id> being a gd level id
  • returns a JSON array of song objects
• /api/v1/get_songs_with_id/<song_id> - <song_id> being a NG/ingame song ID
  • returns a JSON array of song objects

a song object looks like this:

{
    "id":"[INTERNAL SFH ID]",
    "name":"",
    "song_name":"[SONG DISPLAY NAME]",
    "song_id":"[SONG ID (ng, string because SFH has replacements for internal songs)]",
    "download_url":"[DIRECT LINK TO DOWNLOAD SONG]",
    "level_id":[OPTIONAL LEVEL ID],
    "state": "[SFH STATE (rated/unrated/mashup/...)]"
}

The API is subject to change, breaking changes will have their API version increased (v1->v2, etc)

feel free to experiment around with the current API and infrastructure, let me know if you'd like anything changed (for convenience, etc) :3

i might implement a /api/v1/get_all_songs endpoint that returns data in the same format as SFH's /songs does.

update: added state field, possible values defined by SFH (ref: songs.js (permalink))

[Fleeym]

Looks good. If you could include the "state" field that SFH has (tells me if the song is a mashup) it would be cool

[RoootTheFox]

@Fleeym state field added and changes deployed :3

[Fleeym]

Aight, this looks good, I'll see what I can cook up with it. I'll probably be able to dev more in the weekend.

[om3ga6400]

we need this

[om3ga6400]

@Fleeym are we getting this soon?

[Fleeym]

soon-ish. sorry, life is getting in the way at the moment and I have little time for modding :(

[RoootTheFox]

thank you for still considering this - just wanted to mention something: the person who PR'd the mod API stuff (@FlafyDev) is working on a separate mod that does something similar (as well as accumulating custom songs from other locations apart from SFH) which might be worth looking into.

[FlafyDev]

thank you for still considering this - just wanted to mention something: the person who PR'd the mod API stuff (@FlafyDev) is working on a separate mod that does something similar (as well as accumulating custom songs from other locations apart from SFH) which might be worth looking into.

Yep! Submitted the mod to the index geode-sdk/indexer#1733 hoping to get it accepted

[Fleeym]

thank you for still considering this - just wanted to mention something: the person who PR'd the mod API stuff (@FlafyDev) is working on a separate mod that does something similar (as well as accumulating custom songs from other locations apart from SFH) which might be worth looking into.

Yep, I know about the mod, I can't integrate it into jukebox because of it using yt-dlp for downloads from youtube. Really cool addon to jukebox though. I like going the route of empowering the API and letting others go ham with ideas, as I myself don't have much time for modding these days.

[om3ga6400]

if only this still existed

[RoootTheFox]

just use Auto NONG at this point

[om3ga6400]

just use Auto NONG at this point

is there sfh integration?

[RoootTheFox]

is there sfh integration?

yes, via my sfh mirror (sfh-rooot)

[Fleeym]

Bump to remind myself that I should start working on this.