Don't require utf-8 source files #511
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@squeek502 when you get a chance can you try out this PR and see if it fixes the encoding issues you were seeing? You'll need a fairly recent build of the Zig compiler. It should also be somewhat faster than before since we're no longer scanning the whole file up front to validate UTF8-ness. |
|
Haven't investigated fully yet, but preliminary results are good. This branch brings the win32-samples-rc-tests results to: up from |
This removes the requirement that source files be UTF-8 encoded for tokenization and preprocessing. Only the parser cares about the source encoding, so we defer any checks until then.
Vexu
approved these changes
Oct 1, 2023
| const slice = p.tokSlice(p.tok_i); | ||
| const view = std.unicode.Utf8View.init(slice) catch { | ||
| try p.errTok(.invalid_utf8, p.tok_i); | ||
| return error.FatalError; |
There was a problem hiding this comment.
This looks like a place for future improvements.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This removes the requirement that source files always be UTF-8 encoded. Identifiers are still required to be UTF-8 encoded; and there is a todo item for char literals to match clang/gcc behavior when the source text of the literal is not UTF-8 encoded.
Some simple testing on my machine shows that skipping the up-front UTF-8 validation is significantly faster for large files - the 8.4 meg amalgamated sqlite3.c source parses in ~400ms on master; and ~340ms on this branch for me.