ProtonUp-Qt Flatpak ver. 2.9.2 - UnicodeDecodeError

[Megatron-X]

Please fill out following when reporting a new bug:

Describe the bug
When I launch ProtonUp-Qt (Flatpak) I keep getting a UnicodeDecodeError. I also tried to launch it in terminal and still get the UnicodeDecodeError. Prior to a couple of days, I didn't have this problem at all and was able to launch it.

To Reproduce
Steps to reproduce the behavior:

1. Launch ProtonUp-Qt either by Icon or Terminal.
2. See Screenshot and Terminal output

Screenshots

Desktop (please complete the following information):

• Platform: PC
• System: Arch Linux
• Version: ProtonUp-Qt 2.9.2
• How did you install ProtonUp-Qt?: GNOME Software

Terminal output

ProtonUp-Qt 2.9.2 by DavidoTek. Build Info: DavidoTek Flathub build.
Python 3.11.9 (main, Nov 10 2011, 15:00:00) [GCC 13.2.0], PySide 6.7.0
Platform: KDE Flatpak runtime 6.7 Linux-6.6.40-1-lts-x86_64-with-glibc2.38
Loading locale es / es_ES
Loaded ctmod GE-Proton
Loaded ctmod Wine-GE
Loaded ctmod Boxtron
Loaded ctmod D8VK (nightly)
Loaded ctmod Kron4ek Wine-Builds Vanilla
Loaded ctmod Lutris-Wine
Loaded ctmod Luxtorpeda
Loaded ctmod Northstar Proton (Titanfall 2)
Loaded ctmod Proton Tkg
Loaded ctmod Proton Tkg (Wine Master)
Loaded ctmod Roberta
Loaded ctmod Steam-Play-None
cat: /etc/lsb-release: No existe el fichero o el directorio
Loaded ctmod SteamTinkerLaunch
cat: /etc/lsb-release: No existe el fichero o el directorio
Loaded ctmod SteamTinkerLaunch-git
Loaded ctmod vkd3d-lutris
Loaded ctmod vkd3d-proton
Loaded ctmod Wine Tkg (Valve Wine Bleeding Edge)
Loaded ctmod Wine Tkg (Wine Master)
Loaded ctmod DXVK
Loaded ctmod DXVK Async
Loaded ctmod DXVK (nightly)
Error: Could not get a list of all Steam apps: 'utf-8' codec can't decode byte 0xfd in position 2170: invalid start byte
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/app/lib/python3.11/site-packages/pupgui2/__main__.py", line 2, in <module>
    main()
  File "/app/lib/python3.11/site-packages/pupgui2/pupgui2.py", line 568, in main
    MainWindow()
  File "/app/lib/python3.11/site-packages/pupgui2/pupgui2.py", line 109, in __init__
    self.update_ui()
  File "/app/lib/python3.11/site-packages/pupgui2/pupgui2.py", line 230, in update_ui
    global_ctool_name: str = get_steam_global_ctool_name(install_loc.get('vdf_dir'))
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/pupgui2/steamutil.py", line 221, in get_steam_global_ctool_name
    d = get_steam_vdf_compat_tool_mapping(vdf.load(open(config_vdf_file)))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/vdf/__init__.py", line 211, in load
    return parse(fp, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/app/lib/python3.11/site-packages/vdf/__init__.py", line 97, in parse
    for lineno, line in enumerate(fp, 1):
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfd in position 2170: invalid start byte

Gamepad error: No gamepad found.
QThread: Destroyed while thread is still running
/app/bin/net.davidotek.pupgui2: línea 3:     3 Abortado                (`core' generado) python3 -m pupgui2 "$@"

[sonic2kk]

(Note: I believe this is distinct from #415)

Can't reproduce on Arch Linux with Flatpak (v2.9.2) or 887702a.

I wonder if this is locale-related? It seems like this could be a bug in the upstream VDF library with handling unicode characters, which may come about from parsing a config.vdf.

I'll try inserting some unicode characters into a VDF file and see if ProtonUp-Qt (or more specifically, the upstream VDF library) can parse it.

[sonic2kk]

From doing some research it appears that the error may actually be pretty much as stated; the format of the file may not be UTF-8. Can you check the format of your config.vdf file? The location may vary but the most common location on Arch Linux is either:

• ~/.steam/root/config/config.vdf for regular Steam
• ~/.var/app/com.valvesoftware.Steam/data/Steam/config/config.vdf for Flatpak Steam (I believe the path is right)

I couldn't convert to a new charset to test but it might be a good idea to check what the encoding of the file is. Any text editor should tell you, similarly file config.vdf from the folder with the file should reference this as well. chardetect on your system should also be able to tell you.

---

But I suspect this issue comes from the file simply not being UTF-8. This is odd because Python's open should be able to determine the encoding, although you can specify encoding=. But we shouldn't need to do this. Maybe Python is defaulting to the locale of the Flatpak environment (probably UTF-8?) but that does not match the config.vdf encoding?

As per the Python open docs

In text mode, if encoding is not specified the encoding used is platform-dependent: locale.getencoding() is called to get the current locale encoding.

We would be using text mode as this is the default (I believe you have to specify something like rb for bytes mode). On my system, locale.getencoding() returns UTF-8. So perhaps the result of locale.getencoding() does not match the config.vdf encoding. The question would become, why is this file not UTF-8 and in what circumstances would it not be UTF-8? It seems unusual.

[Megatron-X]

From doing some research it appears that the error may actually be pretty much as stated; the format of the file may not be UTF-8. Can you check the format of your config.vdf file? The location may vary but the most common location on Arch Linux is either:

* `~/.steam/root/config/config.vdf` for regular Steam

* `~/.var/app/com.valvesoftware.Steam/data/Steam/config/config.vdf` for Flatpak Steam (I believe the path is right)

I couldn't convert to a new charset to test but it might be a good idea to check what the encoding of the file is. Any text editor should tell you, similarly file config.vdf from the folder with the file should reference this as well. chardetect on your system should also be able to tell you.

But I suspect this issue comes from the file simply not being UTF-8. This is odd because Python's open should be able to determine the encoding, although you can specify encoding=. But we shouldn't need to do this. Maybe Python is defaulting to the locale of the Flatpak environment (probably UTF-8?) but that does not match the config.vdf encoding?

As per the Python open docs

In text mode, if encoding is not specified the encoding used is platform-dependent: locale.getencoding() is called to get the current locale encoding.

We would be using text mode as this is the default (I believe you have to specify something like rb for bytes mode). On my system, locale.getencoding() returns UTF-8. So perhaps the result of locale.getencoding() does not match the config.vdf encoding. The question would become, why is this file not UTF-8 and in what circumstances would it not be UTF-8? It seems unusual.

Thank you for your response.
First, I tested it with vim and, strangely, the only text that appears is "InstallConfigStore". Then I tried it with GNOME's Text Editor and was able to see the text file normally. Finally, I did the chardetect and this is what came up:
config.vdf: ISO-8859-1 with confidence 0.73

[sonic2kk]

Hmm, it sounds like something is wrong with your config.vdf. What size is the file? I think it should be in the region of about 200kb give-or-take depending on various factors (my file is around 350kb but I have a lot of entries in my CompatToolMapping and a lot of games in my library).

The text "InstallConfigStore" would be expected to show up in the file. In fact it should be the first entry in the file. The text-based VDF format is very similar to JSON, with the structure being like this (just a snippet of the first few lines, these sections get closed later on).

"InstallConfigStore"
{
	"Software"
	{
		"Valve"
		{
			"Steam"
			{

If your file size is more than a few bytes (the text "InstallConfigStore" probably only takes up about 40 bytes or less) then I would say when viewing the file it can only display this as the first part of the file. But if that is the entire size of the file, then I suspect there's a problem somewhere!

For what it's worth, ProtonUp-Qt might've crashed if it tried to read this file anyway. I can't remember how strict we are about the contents of config.vdf, we did get stricter recently but I'm not sure how much we verify the format.

---

Some other things that might noteworthy are:

• Does the file modified date line up with the last time you opened / closed Steam? This file gets modified each time the Steam Client is opened and closed.
• If you open Steam, does this file get re-created and look correct afterwards?

[Megatron-X]

Hmm, it sounds like something is wrong with your config.vdf. What size is the file? I think it should be in the region of about 200kb give-or-take depending on various factors (my file is around 350kb but I have a lot of entries in my CompatToolMapping and a lot of games in my library).

The text "InstallConfigStore" would be expected to show up in the file. In fact it should be the first entry in the file. The text-based VDF format is very similar to JSON, with the structure being like this (just a snippet of the first few lines, these sections get closed later on).

"InstallConfigStore"
{
	"Software"
	{
		"Valve"
		{
			"Steam"
			{

If your file size is more than a few bytes (the text "InstallConfigStore" probably only takes up about 40 bytes or less) then I would say when viewing the file it can only display this as the first part of the file. But if that is the entire size of the file, then I suspect there's a problem somewhere!

For what it's worth, ProtonUp-Qt might've crashed if it tried to read this file anyway. I can't remember how strict we are about the contents of config.vdf, we did get stricter recently but I'm not sure how much we verify the format.

The file size is 34,1 kb.

Some other things that might noteworthy are:

* Does the file modified date line up with the last time you opened / closed Steam? This file gets modified each time the Steam Client is opened and closed.

* If you open Steam, does this file get re-created and look correct afterwards?

The file does get modified when I open and close Steam. The file size changes to 34,0 kb.

[sonic2kk]

The file size is probably fine, maybe the average size is smaller than I anticipated 😅 But approx 34kb surely suggests the file stores more than just "InstallConfigStore", and I imagine it just isn't getting displayed.

I did a bit of digging and it seems that, particularly on Arch where the locale is often expected to be configured by the user, you can configure your locale to be non-UTF-8. So this is something to check; can you type locale in your terminal and see if your locales, in particular the LANG variable (should be the first one) ends with UTF-8 or ISO-8859?

You can see more about the locales on Arch on the Arch Wiki Locale page.

I imagine that this is a locale-related issue, and that Python is trying to use UTF-8 when opening the file, but Steam is using an alternative locale and trying to use ISO-8859.

---

From a development perspective, I'm not sure if this is something ProtonUp-Qt should handle. On one hand, Steam appears to respect the system locale. On the other hand, Python should be doing that but it appears that it still tries to use UTF-8 to open the file. I wonder if there's a way we can get a file locale and then pass that to open()... This won't be 100% foolproof (there is no way to be 100% confident in a charset to my knowledge) and there may not even be a good way to do this from Python across distributions so if this is locale-related, perhaps chalking this up to system configuration is fine? I'm not sure if there's a way we should handle this on our end.

[Megatron-X]

The file size is probably fine, maybe the average size is smaller than I anticipated 😅 But approx 34kb surely suggests the file stores more than just "InstallConfigStore", and I imagine it just isn't getting displayed.

I did a bit of digging and it seems that, particularly on Arch where the locale is often expected to be configured by the user, you can configure your locale to be non-UTF-8. So this is something to check; can you type locale in your terminal and see if your locales, in particular the LANG variable (should be the first one) ends with UTF-8 or ISO-8859?

You can see more about the locales on Arch on the Arch Wiki Locale page.

I imagine that this is a locale-related issue, and that Python is trying to use UTF-8 when opening the file, but Steam is using an alternative locale and trying to use ISO-8859.

From a development perspective, I'm not sure if this is something ProtonUp-Qt should handle. On one hand, Steam appears to respect the system locale. On the other hand, Python should be doing that but it appears that it still tries to use UTF-8 to open the file. I wonder if there's a way we can get a file locale and then pass that to open()... This won't be 100% foolproof (there is no way to be 100% confident in a charset to my knowledge) and there may not even be a good way to do this from Python across distributions so if this is locale-related, perhaps chalking this up to system configuration is fine? I'm not sure if there's a way we should handle this on our end.

My locale is: LANG=es_ES.UTF-8
I'm wondering if removing # from en_US.UTF-8 UTF-8 (to use as fallback as stated on the Arch Wiki Locale) and regenerate locale will help... 🤔

[sonic2kk]

It might be worth trying to enable a UTF-8 fallback. I'm not too sure of all the specifics around this, but it does seem odd to me that Steam is writing the file as what is guessed to be ISO-8859, and that this also happens to be a common non-UTF-8 locale.

It seems Python is assuming UTF-8 when that is clearly not the case, and Python should be making this determination based on your system locale settings. We already know that the Flatpak is returning UTF-8 (based on the error causing this issue) but it might be useful to check that your system Python is using, assuming it is installed (iirc Steam at least relies on Python, so if you're not using Flatpak Steam you probably have Python). You can run python3 -c "import locale;print(locale.getencoding())" to execute the Python command to get the encoding it will default to when opening files (-c is a flag you can give to Python to pass a string as a command instead of running an interactive session and exiting).

If this also returns UTF-8, and we know the Flatpak returns UTF-8, and your system is configured to use UTF-8 as you've shown, then I guess for some reason Steam is not using this.

[Megatron-X]

I must apologize that I didn't realize that I missed typing the Flatpak version of Steam which I am using.

I think I've fixed it thanks to you. I removed # from [email protected] (can be done with any ISO) from /etc/locale.gen and then in /etc/locale.conf I added LC_CTYPE=es_ES@euro (or whichever ISO you choose after LC_CTYPE=). Then I did sudo locale-gen and rebooted. After rebooting I launched Steam (Flatpak) and closed it. Then I launched ProtonUp-Qt and it worked like before.

I do not understand why Steam chose to use ISO instead of UTF-8. That's strange... I hope they change to UTF-8.

Thank you!

[sonic2kk]

I'm not sure why Steam uses ISO either, but I did check and I have both the ISO and UTF-8 locale enabled on my system. My install is from a long time ago so I'm not sure why I have it set up this way. I'm glad it's resolved though!

Maybe you should report this upstream, even just to ask and maybe get a clearer answer on what is going on? That should clear up whether it's expected that both locales are enabled, and/or if it is expected that Steam should not use UTF-8. To me it isn't clear from reading around that both locales should be enabled, but I also had both enabled, so I'm not sure if I read something a long time ago when I set up this installation 😅 I'm not sure what my other devices have, but I would wager since ProtonUp-Qt works fine on those devices that they have both locales set up as well (if that is indeed the root cause). I would be interesting to know what the Steam Deck uses, since at launch it had complaints about misconfigured locales that appear to be resolved now.

If you want to report it (although you don't have to!) you're probably best reporting it on the Steam Flatpak repo for now and reporting further upstream to Valve if this is a general thing with Steam that you'd like to see changed.

It is worth noting that the Steam Flatpak is entirely unofficial, and it is known to have a few oddities. Valve do label issues on the Steam for Linux issue tracker when they are experienced with Steam Flatpak, but to my understanding they only directly develop and maintain the Steam .deb which other distros' repos build off of (including Valve themselves on SteamOS).

[Megatron-X]

After reading the Steam Wiki from Arch Linux carefully https://wiki.archlinux.org/title/Steam I didn't realize that in the Installation section under "In order to run Steam on Arch Linux:" it states:

• If not already done during installation, you must generate the en_US.UTF-8 locale to prevent invalid pointer errors.

So I proceeded to uninstall Steam Flatpak and eliminate the directory (games are on another SSD) and redid the locale and locale-gen installation without the ISO and only the UTF-8 of both en_US (as fallback) and es_ES to see if it works. After rebooting and install Steam Flatpak, I closed it and quickly launched ProtonUp-Qt to see if it will work or not. It worked. I even tried out the command you told me to run chardetect config.vdf in ~/.var/app/com.valvesoftware.Steam/data/Steam/config/ now it comes out as config.vdf: ascii with confidence 1.0. Now ProtonUp-Qt works without any issues!
The locale of en_US.UTF-8 was the missing link all along which I didn't have before. I didn't think it would be necessary for the Flatpak version of Steam either, but it's needed.
Thank you @sonic2kk for your time, patience, and help with this. 👍👍👍

[sonic2kk]

No problem at all, thank you for the follow-up here! I didn't realise this myself either (I guess I had this a long time ago? Unsure, it has been many years since i did this installation so I'm not sure what past-sonic2kk did 😅) but those steps sound reasonable and that seems like a very little-known step. Particularly if you're not using an en_US locale it would not be obvious to me to include that.

Hopefully if anyone hits this in future this can bring a bit of visibility to the issue, so thank you as well 😄

[DavidoTek]

Interesting, learned something new again.

Hopefully if anyone hits this in future this can bring a bit of visibility to the issue, so thank you as well 😄

Good to hear it is fixed, thanks for helping! I will close this issue.

[ben-novanus]

I also have this issue, I'm on Manjaro. My flatpak install of protonup was working fine and then it just stopped working a couple of weeks ago. I followed the directions and found that all of my locales were still configured correctly with en_US.UTF-8

[sonic2kk]

I don't think after discussion that this is a ProtonUp-Qt issue but still boils down to some kind of system configuration issue. It might be good to ask on the Manjaro forums and see if they can help troubleshoot what is likely still some kind of locale issue.

Perhaps Valve have changed something here recently? I'm not sure, I never had any issues like this with my ProtonUp-Qt installation.

It would also be interesting if you can check if this happens with the ProtonUp-Qt 2.10.2 AppImage, just in case some updated dependencies fix the issue.

[ben-novanus]

Same issue with the v2.10.2 appimage FYI.

[Karfumble]

Same issue on Linux Mint 21.2

[DavidoTek]

A similar issue was reported for the vdf dependency back in 2019 and was fixed by replacing .decode('utf-8') with .decode('utf-8', 'replace'): ValvePython/vdf#20

import vdf
config_vdf_file = "~/.steam/root/config/config.vdf"  # or similar

# OLD CODE
data = vdf.load(open(config_vdf_file))

# SUGGESTION
with open(config_vdf_file, "r", encoding="utf-8", errors="replace") as f:
    data = vdf.loads(f.read())

Can someone test the code snippets with their corrupted config file and share the results?

[ben-novanus]

@DavidoTek that is the fix

In file ProtonUp-Qt/pupgui2/steamutil.py
By changing line 221 from

d = get_steam_vdf_compat_tool_mapping(vdf.load(open(config_vdf_file)))

to

with open(config_vdf_file, "r", encoding="utf-8", errors="replace") as f:
    data = vdf.loads(f.read())
d = get_steam_vdf_compat_tool_mapping(data)

It does load the application and it appears to work. But then the game list will be empty and in the console it will show this many times:
Error: Could not get a list of all Steam apps: 'utf-8' codec can't decode byte 0xb9 in position 5833: invalid start byte

[DavidoTek]

Thanks for testing! Now we are a step further.

I haven’t checked that yet but I think the error still appears because there are multiple places in the code where vdf.load is used.

We should add a function like steamutil.py#vdf_safe_load(file_path) and replace vdf.load with it everywhere.