implement qs8 x8c4 pack using avxvnni

[kylo5aby]

qs8_packw/xnn_qs8_packw_gemm_goi_ukernel_x8c4__scalar_sd1x_encoder_decoder/B:1/M:512/N:4096/K:4096/real_time 3220123 ns 2962085 ns 211 bytes=10.4215G/s elements=5.21012G/s
qs8_packw/xnn_qs8_packw_gemm_goi_ukernel_x8c4__avxvnni_sd1x_encoder_decoder/B:1/M:512/N:4096/K:4096/real_time 1534454 ns 710227 ns

qs8_packw/xnn_qs8_packw_gemm_goi_ukernel_x8c4__scalar_sd1x_diffusion/B:8/M:64/N:64/K:160/real_time 15215 ns 13587 ns 43699 bytes=10.8019G/s elements=5.38414G/s
qs8_packw/xnn_qs8_packw_gemm_goi_ukernel_x8c4__avxvnni_sd1x_diffusion/B:8/M:64/N:64/K:160/real_time 5801 ns 1592 ns 117772 bytes=28.33G/s elements=14.1209G/s

qs8_packw/xnn_qs8_packw_gemm_goi_ukernel_x8c4__scalar_attention/B:7/M:1/N:16/K:2304/real_time 48944 ns 47492 ns 13489 bytes=10.5469G/s elements=5.27231G/s
qs8_packw/xnn_qs8_packw_gemm_goi_ukernel_x8c4__avxvnni_attention/B:7/M:1/N:16/K:2304/real_time 14653 ns 7623 ns 45095 bytes=35.2285G/s elements=17.6104G/s

[fbarchard]

if you have time, try some different offsets here and run on meteorlake, alderlake or lunar lake.
448 is 7 cache lines of 64 and a good choice
960 is 15 cache lines of 64 and may work better on large packw
128 is 2 cache lines ahead and sometimes good when doing a small amount of loads. You are reading 32 bytes, so these prefetches are somewhat redundent.
4096 is interesting due to page

instead of prefetching the same cachelines, it might be worth the logic to only prefetch half the time

[kylo5aby]

I have tried those different offsets on MTL on benchmark qs8_packw_bench, and 960 has the lowest cache miss, but 448 has better performance. for alternating prefetch, instructions is reduced, but totally there is no performance improvement.

[fbarchard]

This loop could be removed. In practice KC is a large value most of the time

[fbarchard]

the prefetch distance should remain constant.

[fbarchard]

unaligned_load_u32 will read out of bounds. I think we need to be asan friendly and remove the 'EXTRA_BYTES' in the calling unittests and stay within bounds.
To keep the source code simple, maybe write a function to replace unaligned_load_u32(w0)); which reads 1 to 3 bytes, passing k.
On avx10 we'll be able to set up a kmask. The same source code will be used, but compiled as avx512

[fbarchard]

add a space after casting.
value |= (uint32_t) bytes[i] << (i * 8);

[fbarchard]

remove XNN_OOB_READS
This kernel should be asan friendly now

[fbarchard]

I think this function should follow the naming conventions of sse or the function it replaces
unaligned_load_u32
becomes
safe_load_u32

[fbarchard]

on a future avx10 version of this kernel, this and the remainder code could use a kmask to read and embedded broadcast to the correct channels.

[fbarchard]

optionally use something like safe_load to read 1 to 7 int32 and then store a full vector.

[kylo5aby]

maybe we can write something like:

XNN_INLINE static __m256i safe_load_avx(const void* src, size_t n) {
    int32_t temp[8] = {0}; 
    for (size_t i = 0; i < n; ++i) {  
        temp[i] = ((const int32_t*)src)[i];
    }
    return _mm256_loadu_si256((const __m256i*) temp);  
}

when n is little, I wonder the memory overhead will be a problem

[fbarchard]

This can store full vectors of 0 the same as the main NC loop

    _mm256_storeu_si256((__m256i*) (out + 0), _mm256_setzero_si256());

[kylo5aby]

that maybe cause memory out of bound? unless add a mask

[kylo5aby]

updated. the output is padded, so it will not out of bound. Thanks the feedback

[fbarchard]

overall looks good

[dsharletg]

This is failing to compile:

XNNPACK/src/qs8-packw/gen/qs8-packw-x8c4-gemm-goi-avxvnni-prfm.c:403 error: variable 'vksum0' is uninitialized when used within its own initialization [-Werror,-Wuninitialized]
  403 |       __m256i vksum0 = _mm256_mullo_epi32(vksum0, vzeropoint);

[kylo5aby]

This is failing to compile:

XNNPACK/src/qs8-packw/gen/qs8-packw-x8c4-gemm-goi-avxvnni-prfm.c:403 error: variable 'vksum0' is uninitialized when used within its own initialization [-Werror,-Wuninitialized]
  403 |       __m256i vksum0 = _mm256_mullo_epi32(vksum0, vzeropoint);

Resolved.