Non-Steam Game Categories Not Working

[seanryanmcewan]

System Information

• SteamTinkerLaunch version: v12.12
• Distribution: SteamOS
• Installation Method: ProtonUp-Qt

Issue Description

Hello! I'm using steamtinkerlaunch for it's ability to add non-steam games to the steam library via command line options.

I am uncertain whether the 'tags' option is currently supposed to be working. This issue from several years ago mentioned that it wasn't working, but then the following comments were unclear as to whether that was resolved or not.

To be clear, I am running the script, closing Steam, then reopening Steam.
SteamTinkerLaunch successfully added the game to the library, but it was not added to any Collections.
I also tried the linked solution of running steam://resetcollections. I rebuilt my collections and tried again, and it still doesn't seem to be working.

Is this intended to be working?
If it's not, perhaps looking at Steam Rom Manager would be helpful - they are somehow successfully adding items to the Steam Library and adding them to collections.

Thanks for reading!

Logs

steamtinkerlaunch.log

[sonic2kk]

Have you tried using the latest SteamTinkerLaunch (SteamTinkerLaunch-git from ProtonUp-Qt with Advanced Mode enabled)? You're using v12.12 which is very old. Also, Add Non-Steam Game got a lot of useful improvements lately. It also correctly sets the Steam AppID now, which may resolve the problem.

I'll take a look and see if tags are working, perhaps how we write them out is wrong or something. This worked before a long time ago, but I'll take a look when I have some time.

If it's not, perhaps looking at Steam Rom Manager would be helpful - they are somehow successfully adding items to the Steam Library and adding them to collections.

That program is written using custom libraries in TypeScript, SteamTinkerLaunch writes out directly to the binary VDF file and is written in Bash, so it is not super helpful unfortunately.

[sonic2kk]

I've confirmed the issue with the latest master (93ec382), but I haven't figured out the cause. I made a few casing changes, and after that, byte-for-byte I can generate the same shortcuts.vdf file from Steam as I can with STL. Tthe file contents are identical, making sure categories are selected in the same order, naming and paths are the same (including quotes and no trailing space), so I can't figure out why the tags in the shortcuts.vdf aren't being read properly.

My hunch then is that Steam sets the collections elsewhere, maybe in the same place it uses for Steam games now, but even when editing categories from Steam, Steam itself updates shortcuts.vdf to have these new tags. So I'm not sure, I guess the next thing to try is to edit a shortcuts.vdf file from Steam using the Python VDF library, write out some new categories, and see if Steam picks them up. If Steam still doesn't pick up the categories, then I think we need to set Steam categories elsewhere, as the tags part of the VDF is not the only piece of the puzzle there.

Also:

Hello! I'm using steamtinkerlaunch for it's ability to add non-steam games to the steam library via command line options.

I'm pretty happy to hear this. This is how I use the ansg command primarily and it's been receiving a lot of improvements lately, and I've kept commandline usage in mind. I think the commandline usage for it is pretty powerful and has lots of use-cases, so I'm happy to know it's not just me using it :-)

[sonic2kk]

Confirmed that writing categories out to the shortcuts.vdf file using Python was not sufficient, categories did not update.

Here is an example script:

import vdf

user_id=''
vdf_path = f'/home/user/.local/share/Steam/userdata/{user_id}/config/shortcuts.vdf'
vdf_dict = vdf.binary_load(open(vdf_path, 'rb'))

# Set first category for first shortcut to 'Valve'
vdf_dict['shortcuts']['0']['tags']['0'] = Valve

print(vdf.binary_dumps(vdf_dict['shortcuts']['0']['tags']))  # Verify new tag was added

# Write out new binary content
with open(vdf_path, 'wb') as new_vdf:
    new_vdf.write(vdf.binary_dumps(vdf_dict))

After opening Steam, the new shortcut is not present, despite it being in the VDF file. Here's a screenshot from Okteta inspecting the raw VDF file of my own shortcuts.vdf with some random tags after doing something similar to the above from a Python shell

---

Steam must store the categories elsewhere for Non-Steam Games.

[sonic2kk]

An investigation suggests it's stored in some leveldb location, uh oh... That seems to only be for new collections and Steam games, Non-Steam Games don't seem to be in any leveldb location (changing shortcuts didn't update any files there, and there is no mention of any AppIDs in this file).

[sonic2kk]

It looks like information on which tags a game is using is stored in ~/.local/share/Steam/userdata/<userid>/config/localconfig.vdf, the same folder as shortcuts.vdf.

I discovered this when I noticed localconfig.vdf was getting updated, so I did a diff with a version of the file before and after. This field was updated:

"user-collections"              "{\"from-tag-Danganronpa\":{\"id\":\"from-tag-Danganronpa\",\"added\":[<appid>],\"removed\":[]},\"from-tag-Chucklefish\":{\"id\":\"from-tag-Chucklefish\",\"added\":[2717785613],\"removed\":[]},\"hidden\":{\"id\":\"hidden\",\"added\":[<appid>],\"removed\":[]},\"uc-1QwE3sdxpu7J\":{\"id\":\"uc-1QwE3sdxpu7J\",\"added\":[<appid>],\"removed\":[]}}"

This is an escaped JSON string, and controls which Collections a Shortcut shows up in, as well as the hidden status! I don't understand yet how collections are inserted or updated, and how SteamTinkerLaunch needs to write out to this file just yet, but it was interesting.

---

I mentioned that the Hidden status of a game is stored in this file as well. If you recall, even though Steam doesn't read collections from shortcuts.vdf at all, it still expects this field to be present, and it still updates this when you insert a new collection. However, this is not the case for IsHidden. You can give this field either \x00 or \x01, but Steam will not read nor update this field. The same goes for AllowOverlay.

In other words, Steam uses localconfig.vdf to track the following values now

• Steam Collections, though it will update shortcuts.vdf
• Hidden, and it will ignore the value in shortcuts.vdf unlike Collections which it does actually update
• AllowOverlay, same as above it ignores the shortcuts.vdf value and solely uses localconfigvdf.
• OpenVR, it will update and respect this flag in shortcuts.vdf as well as localconfig.vdf.

For AllowOverlay, it actually stores this in a separate block to user-collections above. It tracks the Steam Overlay flag and the OpenVR flag in localconfig.vdf, but it actually respects the setting in shortcuts.vdf for the OpenVR flag and updates it. This block in localconfig.vdf is under the "Apps" section of the file, and for Non-Steam Games, it uses the signed 32bit integer AppID instead of the standard unsigned 32bit AppID. The block looks like this:

"Apps"
{
    # ...

    "-804838568"
    {
        "OverlayAppEnable"		"0"
        "DisableLaunchInVR"		"1"
    }
}

---

Therefore, on the SteamTinkerLaunch side, we need to make the following changes:

• Figure out how to write out/update user-collections in shortcuts.vdf. Likely, we can grep this and use JQ to edit the entry, then remove the original value from localconfig.vdf and write out a new entry containing updated collection information, basically appending to the existing information in the file (if present) and generating a new string to write out.
• The above also applies for setting a shortcut as Hidden.
• For AllowOverlay and OpenVR, we should still update shortcuts.vdf even if it is ignored by Steam for AllowOverlay, and then figure out how to update the block in localconfig.vdf to write out the correct values.

That'll be a pretty significant refactor, but hopefully doable.

[sonic2kk]

It looks like the user-collections part of the VDF file stores information even about shortcuts which were removed, so the valid, parsed JSON would look like this, for a shortcut with two collections (copied the text directly from localconfig.vdf, did JSON.stringify(JSON.parse('string')) on it, then formatted it in a text editor):

{
    "uc-1QwE3sdxpu7J": {
        "id": "uc-1QwE3sdxpu7J",
        "added": [
            <appid>
        ],
        "removed": []
    },
    "uc-y5nlF+rzO*+fR": {
        "id": "uc-y5nlF+rzO*+fR",
        "added": [
            <appid>
        ],
        "removed": []
    }
}

I assume "uc-<blah>" is some type of encoding for the category name, but I haven't figured it out yet. I also assume that the collections store information about which AppIDs are stored in them, not the other way around, which may make the manipulation of this more tricky. In other words we'll have to check if the collection is mentioned in this object, and if it is, add our AppID into it, otherwise create a new entry.

It's worth noting too that <appid> is stored as an integer, again the unsigned 32bit integer.

However even if these invalid entries are removed (i.e. collections added for shortcuts no longer in the library), Steam will remember them and re-insert them. So it must track it somewhere else, perhaps in a LevelDB file. So this makes me wonder if inserting a new collection here will actually work, or if this is just a "cache" that actually pulls from LevelDB...

I hope not, because updating information under "Apps" for the Overlay and VR settings does actually work, so I hope the same applies for collections.

[sonic2kk]

Another complication: Not all tags use this encoded format. We sometimes have to use a different format, where we can insert an entry for a new tag like this:

"from-tag-tagname": {
    "id": "from-tag-tagname",  // Must match object name
    "added": [
        <appid>
    ],
    "removed": []  // This can always be blank
}

I noticed this because for some collections, Steam uses this format, but not for others. For example, my "ATLUS" tag uses uc-1QwE3sdxpu7J, but my "Final Fantasy" tag (which is considerably newer), uses from-tag-tagname.

I don't know if it's possible to know which tag will use which format. Perhaps whatever say SteamTinkerLaunch parses information about available tags will give us a clue, but I haven't investigated deeply yet.

---

My guess is tools like Steam-ROM-Manager don't have this problem because they create new categories, and expect the category names they provide to not exist.

---

Despite all of this, it seems like we can insert collections this way. We just need to figure out the logic to insert, and also figure out a way to know which tag format to use (the encoded uc-<blah> or from-tag-tagname).

[sonic2kk]

It looks like we can find out if a tag uses the old-style or new-style tag naming convention by parsing ~/.local/share/Steam/config/htmlcache/Local Storage/leveldb/006479.log This is a binary log from LevelDB, and I think this is consistent across devices though I'll need to confirm. We could always fall back to trying to grep from the ldb file directly from some known bytes.

If we can figure out how Steam encodes the collection names, we could take an input collection name - say "Shortcuts" - and convert it to this format. Then we can check if the file has either user-collections.Shortcuts or the encoded version like user-collections.q12kj4kjr or whatever. There should only be one.

Depending on which we get a match for, we can then infer which name to use when writing out to user-collections in the localconfig.vdf.

The tricky part now is figuring out how these strings are encoded.

[sonic2kk]

Actually, for which tag name is used, it seems to be the other way around: from-tag-tagname is for older tags, and uc-<encoded> is for newer collections.

Still no lads for how these are encoded, I can't discern a pattern because even tags starting with the same letters look entirely different. For example "A" and "AA" are both entirely different tag patterns. Instead, it seems to be some form of ID. For example, the following are two collections created about 15 seconds apart. The first one is "A", and the second one is "B" (despite how the client displays collections, they are stored case-sensitive, which can be seen in ~/.local/share/Steam/userdata/<id>/7/remote/sharedconfig.vdf):

{
    "uc-9DEqXUO*+qs06": {
        "id": "uc-9DEqXUO*+qs06",  // "A"
        "added": [
            <id>
        ],
        "removed": []
    },
    "uc-38QEEdaMgmw0": {
        "id": "uc-38QEEdaMgmw0",  // "B"
        "added": [
            <id>
        ],
        "removed": []
    }
}

We may have to resort to grepping this from some LevelDB file or some other file...

Possibly relevant Steam-ROM-Manager file: https://github.com/SteamGridDB/steam-rom-manager/blob/cef009ef9b3bd742052a4c968747b180d89ea564/src/lib/category-manager.ts (although it looks like they write directly to LevelDB)

[sonic2kk]

There is one LevelDB file that has the structure we need. It's a binary file but it has a list of lots of information including information about Steam collections. We can find this file by grepping for the known JSON start hex bytes from all files in the ~/.local/share/Steam/config/htmlcache/Local Storage/leveldb folder, once we find it, turn the file into hex with xxd, then grep for the known start bytes and end bytes, convert it back to decimal, and parse it with jq.

This is a quick-and-nasty command I used in my terminal to write the grepped JSON out:

for f in $(find . -type f); do
    if cat "$f" | xxd -p -c 0 | grep -qoP "636c6f75642d73746f726167652d6e616d6573706163652d31969806015b5b"; then
        cat "$f" | xxd -p -c 0 | grep -oP "636c6f75642d73746f726167652d6e616d6573706163652d3196980601\K.*?(?=01495f68747470733a2f2f737465616d6c6f6f706261636b2e686f73740001)" | xxd -r -p > "collectionsjson.json"
        break
    fi
done

This will produce the JSON we need. Sadly, some characters appear to be invalid, especially once we go beyond the user collections. We'll need to figure out a way to resolve the invalid characters when converting back to decimal with xxd, or perhaps using sed to remove the invalid characters.

Once the JSON is sanitised though, we can parse it with something like this cat "collectionsjson.json" | jq '.[][1] | select( .value != null ) | .value | fromjson | [.id, .name]' -- Though this still isn't perfect, once it hits an object that doesn't have .id, it dies, because this selection also includes showcase objects etc.

[sonic2kk]

The contents of this file may not always be reliable, for example if it is parsed while Steam is writing into it, or while Steam is closing. However, in a happy-path scenario, the following script can fetch the collection ID by collection name:

#!/usr/bin/env bash

## get_steam_collection_id.sh
##
## Example usage: bash get_steam_collection_id.sh "Danganronpa"
## Example output: Danganronpa -> uc-k4365+6f3 

startbytes_nobrackets="636c6f75642d73746f726167652d6e616d6573706163652d31.*?"
startbytes_withbrackets="${startbytes_nobrackets}5b5b"
endbytes="01495f68747470733a2f2f737465616d6c6f6f706261636b2e686f73740001"

steam_leveldb_path="$HOME/.steam/root/config/htmlcache/Local Storage/leveldb"

jsonfilename="${steam_leveldb_path}/collectionsjson.json"
findname="$1"

if [ -f "${jsonfilename}" ]; then
    rm "${jsonfilename}"
fi

# grep on hex representation of each file in the leveldb dir
# for known byte sequence representing collections json
for f in "$(find "${steam_leveldb_path}" -name "*.log" -type f)"; do
    filebytes="$( xxd -p -c 0 "$f" )"
    if grep -qoP "${startbytes_withbrackets}" <<< "${filebytes}"; then
        # Parse the JSON bytes out based on known start and end bytes,
        # convert to char which should be valid JSON, and write out to file
        grep -oP "${startbytes_nobrackets}\K.*?(?=${endbytes})" <<< "${filebytes}" | xxd -r -p | strings | tr -d '\n' > "${jsonfilename}"
        break
    fi
done

# If JSON file wasn't created, no file matching bytes above was found
if [ ! -f "${jsonfilename}" ]; then
    echo "Could not find LevelDB file with JSON collection information, try closing and re-opening Steam"
    exit
fi

# Use JQ to parse out the 'value' property from each JSON object starting with 'user-collections' 
# Each collection entry has a 'value' property which is a JSON string that we can parse out
# and get the 'id' and 'name' properties from
while read -r CATENTRY; do
    CATNAME="$( echo "${CATENTRY}" | jq -r '.name' )"
    CATID="$( echo "${CATENTRY}" | jq -r '.id' )"

    if [ "${CATNAME}" = "${findname}" ]; then
        echo "${CATNAME} -> ${CATID}"
        break
    fi
done <<< "$( jq -r '.[][1] | select(.key | startswith("user-collections")) | select (.value != null) | .value | fromjson | tojson' "${jsonfilename}" )"

This is really fragile though, and if parsed at the wrong time, will miss categories or be entirely unreadable by jq. So even though this can work, it isn't reliable enough yet imo.

[sonic2kk]

Hmm, in testing, this seems to only be inaccurate while Steam is starting up or shutting down. Occasionally it breaks when Steam is running.

I will need to do much more testing but this could be incorporated into STL with the caveat that Steam must be closed before we can write out collections.

I considered storing and parsing collections, but I think fetching from Steam is probably generally ok if we don't try to add a collection while Steam is running or shut down.

[sonic2kk]

Tinkering around with making a tiny C++ program that can parse the LevelDB and interestingly I'm running up against a lot of the same formatting challenges as I was with parsing the log file. The data is reliably able to be extracted (unless Steam is running, because the DB is locked). I can sanitise it with Bash mostly I think (JQ doesn't like it but I'm still not seeing any errors) but I'm not sure how to sanitise it with C++...

[sonic2kk]

I should note that there are still various challenges with this approach:

• We can statically link LevelDB but the executable size may go up
• No idea about licensing issues, since LevelDB is licensed under BSD Clause-3...
• This is untested outside of my Steam account, but I wrote the program to be flexible enough
• The user will have to be online before they can add tags, because they cannot be inserted correctly without this program to get the tags JSON
• We still have to parse the JSON which may take time, though JQ is pretty freaking quick
• This dependency will have to be manually grabbed by STL which some users may be uncomfortable with (some users may not like that a program using a Google dependency has to be used by STL, for example, though Steam also has many Google dependencies including Chromium so... eh)

I think the main way to get around this is just transparency on the wiki.

[sonic2kk]

Note to self: If we can't statically link against LevelDB, we could get it from the Arch mirrors for SteamOS: https://archive.archlinux.org/packages/l/leveldb/

Though I'd prefer something more self-contained.

[sonic2kk]

Created the little C++ program, it's up on GitHub at sonic2kk/DumpSteamCollections. There are no releases, and maybe just dynamically linking to leveldb is fine (though we'll have to consider SteamOS if it doesn't have leveldb).

EDIT: Nope, SteamOS doesn't have LevelDB. It doesn't come as a standard binary, like innoextract, but I wonder if we can simply download and extract it to our Steam Deck install deps folder (the folder structure is the same) since we export that onto PATH.

It is somewhat unfortunate to introduce this as a dependency for adding Non-Steam Games, but leveldb is smaller than jq, which will be a dependency anyway, so it's probably not a big deal. Avoiding any static-linking should also save us from any licensing headaches.

[sonic2kk]

The overall binary size using -O2 optimisation and strip seems to be around 190kb. I would prefer smaller (ideally sub-100k, but I doubt that's really feasible.

[sonic2kk]

Did some tinkering and removed some headers, binary size is down to just over 30kb. Pretty happy with that tbh, even with LevelDB and its snappy dependency, that still adds just under 600kb of a dependency (about 580kb).

Speaking of this dependency, I am unsure if the Steam Deck has it. If it doesn't, we will need to download that, which will be a real pain... I'm wondering if we could build an AppImage for this little program that includes these two dependencies. I'm unsure with how that would play with licensing though. Perhaps it's just something to document on the program's readme, and then on the STL wiki.

I have no idea how to create an AppImage though, or what that might mean for file size implications (hopefully very little, since we only need to include a couple of dependencies for these executables).

[sonic2kk]

Steam Deck does indeed come with snappy, but AppImage might still be worth exploring. It would mean we don't have to download leveldb from the Arch Mirrors.

[sonic2kk]

Using DumpSteamCollections, we can probably attempt to store a cache of collections somewhere. This means we always have some collections available. Since we can't read them when Steam is running, we'll just store what we can find and display those as available collections.