Skip to content
This repository has been archived by the owner on May 13, 2024. It is now read-only.

Fix character encoding conversion issues #249

Closed
wants to merge 5 commits into from

Conversation

grorp
Copy link
Member

@grorp grorp commented Oct 5, 2023

Continuation of #221. Fixes #216 and fixes luanti-org/luanti#13646.

The Irrlicht string conversion functions are not broken, they are just the wrong kind of conversion functions: Irrlicht uses wide <-> multibyte conversion functions in places where it should actually use wide <-> UTF-8 conversion functions. This usually works because the locale-defined multibyte encoding is UTF-8 in many environments, but sometimes, it doesn't.

This PR introduces new wide <-> UTF-8 conversion functions, utf8ToWString and wStringToUTF8, and uses them where appropriate. wStringToMultibyte is not used anymore and has been removed, multibyteToWString has been kept because it is still used in one place.

EDIT: btw, "AutomatedTest" doesn't even compile on Windows.

To do

This PR is Ready for Review. Before merging, 69622fe should be reverted.

How to test

Compile Minetest on Windows with and without SDL. Set a weird combination of language settings. Verify that Unicode text is displayed/copied/whatever-ed correctly.

@grorp grorp added the bug Something isn't working label Oct 5, 2023
@grorp grorp requested a review from sfan5 October 5, 2023 16:20
{
const u32 s = source ? (u32)wcslen(source) : 0;
return wStringToMultibyte(destination, source, s);
return utf8ToWString(destination, source.c_str());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can a stringc contain embedded nulls? If so, this shouldn’t delegate but use another from_bytes:
conv.from_bytes(source.begin(), sorce.end())

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to suggest it can't:

//! verify the existing string.
string<T>& validate()
{
// truncate to existing null
u32 len = calclen(c_str());
if (len != size())
str.resize(len);
return *this;
}

@@ -142,7 +144,7 @@ endif()
# OpenGL

if(USE_SDL2)
option(ENABLE_OPENGL3 "Enable OpenGL 3+" TRUE)
option(ENABLE_OPENGL3 "Enable OpenGL 3+" FALSE)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase mistake?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I temporarily disabled the new driver to make CI pass: 69622fe
I'll revert that commit and rebase this PR now that #250 is merged.

@sfan5
Copy link
Member

sfan5 commented Oct 15, 2023

Merged.

EDIT by grorp: as commits 93eebed and c766c3a

@sfan5 sfan5 closed this Oct 15, 2023
@grorp
Copy link
Member Author

grorp commented Oct 16, 2023

Thanks!

@grorp grorp deleted the fix-encoding-conversion branch December 18, 2023 16:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Windows: Unicode text is garbled when copied to clipboard SDL 2 on Windows: Problems with Unicode/UTF-8
3 participants