[stdlib] Micro-optimize utf8 helper functions

[martinvuyk]

Micro-optimize utf8 helper functions

[JoeLoser]

@ConnorGray do you mind taking a look at this since you've been overhauling how we expose the UTF-8 helpers, Char, and friends?

[martinvuyk]

Hi @JoeLoser and @ConnorGray I'm back from vacation and had some time today to look over the new Char type. FWIW I think it's a good initiative, but IMO it has to be renamed to UTF32Char or something along those lines. It is expensive to create when iterating through long chains of text encoded in a different format (UTF8, UTF16). So IMO we need to create the equivalent UTF8Char and eventually if needed UTF16Char.

Each encoding has its own tricks that are faster for their respective raw data format. Transforming to and from utf32 to then check a byte is wasteful. Going UTF32 is inefficient for mostly ASCII text as well. It is also confusing for people why Char is different from List[Byte] (in the case of UTF8Char the underlying storage could be List[Byte] or (if we have a dynamically owned/CoW buffer one day) Variant[Span[Byte], List[Byte]]).

I was focusing on building out our UTF8 capacity to its maximum with all the little tricks I could find. Then start thinking about whether we want to support UTF32 fully (sidenote: Python is migrating over to UTF8 anyway). IMO the Char type is a nice to have that rounds up all the helper functions I've been tinkering with since I started contributing last year, but there is a lot to build out before I feel comfortable committing to making it a builtin type. UTF32 is not even our main String encoding to make Char use it by default, and like I said when doing some high perf stuff with UTF8 processing like #3528 there is a lot of need for every optimization.

What I propose: Let us build UTF8Char and also rename Char to UTF32Char since many conversions to and from Unicode are needed as evidenced by the amount of helper functions involved. Have both be inside the collections/string sub-package and not as builtins. Then we can insert all utf8 and utf32 helpers that are dispersed in string_slice.mojo into them and try and setup a readable API. Then (months later) we can try and unify both in a single trait.

Edit: I opened a full/better thought through proposal #3988

[ConnorGray]

!sync

[modularbot]

✅🟣 This contribution has been merged 🟣✅

Your pull request has been merged to the internal upstream Mojo sources. It will be reflected here in the Mojo repository on the main branch during the next Mojo nightly release, typically within the next 24-48 hours.

We use Copybara to merge external contributions, click here to learn more.

[modularbot]

Landed in bdaca0f! Thank you for your contribution 🎉