use `ato_simd` for fast &[u8] to int conversion #1423

jqnatividad · 2023-11-18T01:25:14Z

faster than std::parse - SIMD accelerated and skips utf8 validation

now that Polars is also using it

- use atoi_simd for integer conversion, skipping utf8 validation altogether - if number is not an int, use SIMD accelerated utf8 checking, and just use std::parse to get float - if not float, just return None

jqnatividad added 6 commits November 17, 2023 20:19

replace atoi with atoi_simd

d66e8ab

now that Polars is also using it

schema: use atoi_simd instead of std::parse

742ffb1

sort: refactor next_num helper function

e8c21b0

- use atoi_simd for integer conversion, skipping utf8 validation altogether - if number is not an int, use SIMD accelerated utf8 checking, and just use std::parse to get float - if not float, just return None

stats: use atoi_simd for int conversion, insted of std::parse

f871427

validate: use atoi_simd instead of std::parse for int conversion

4bd214d

rustfmt

d3b5747

jqnatividad merged commit b5fae59 into master Nov 18, 2023
3 of 4 checks passed

ZhongRuoyu mentioned this pull request Nov 20, 2023

qsv 0.119.0 Homebrew/homebrew-core#154879

Merged

jqnatividad deleted the atoi_simd branch December 12, 2023 05:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use `ato_simd` for fast &[u8] to int conversion #1423

use `ato_simd` for fast &[u8] to int conversion #1423

jqnatividad commented Nov 18, 2023

use ato_simd for fast &[u8] to int conversion #1423

use ato_simd for fast &[u8] to int conversion #1423

Conversation

jqnatividad commented Nov 18, 2023

use `ato_simd` for fast &[u8] to int conversion #1423

use `ato_simd` for fast &[u8] to int conversion #1423