[RFC] Multithread packing #7545
Conversation
|
@alankelly @dsharlet @fbarchard Putting up an RFC for multi threading packing at subgraph create. Please take a look at the commits labeld [MultiThreaded Packing][RFC]. |
|
see around ~4.5x perf wins on first time model load. |
| @@ -0,0 +1,9 @@ | |||
|
|
|||
| // Copyright 2024 Google LLC | |||
| size_t extra_bytes_bl, | ||
| size_t extra_bytes_n, |
There was a problem hiding this comment.
why do we need these from the API point of view?
src/packw.c
Outdated
| /*extra_bytes=*/context->extra_bytes_bl, | ||
| /*extra_bytes_n=*/context->extra_bytes_n, | ||
| /*params=*/context->params); | ||
|
|
| const uint8_t* w0 = (const uint8_t*) weights; | ||
| const uint16_t* s0 = (const uint16_t*) scale; | ||
| size_t n = nc; | ||
| for (;n >= ${NR}; n -= ${NR}) { |
|
|
||
| // KC/2 bytes is KC Nibbles | ||
| $for N in range(1, NR): | ||
| const uint8_t* w${N} = w${N-1} + (kc >> 1); |
There was a problem hiding this comment.
this kernel does not support odd KC?
how do we ensure convolutions dont call this when kc is odd?
for x4-pack scalar kernel it still supports any KC, inefficiently
then from avxvnni I check if KC is odd and call the scalar kernel to handle it
There was a problem hiding this comment.
This kernel isn't used for convolutions, and only for fully connected layers.
A constraint applied on blockwise quantization is that kc must be divisible by blocksize, and block size is a multiple 32.
There was a problem hiding this comment.
code pointer? Let's make sure they are not just asserts and we fail with proper error message if not doing that already.
mcr229
force-pushed
the
multithread_packing
branch
from
bbb8982 to
eb8384b
Compare
mcr229
force-pushed
the
multithread_packing
branch
from
eb8384b to
30bd2ad
Compare
Enable Multithreaded packing routines in xnn_create. This allows us to multi thread our packing routines at initialization, which can help with more performant first time model loads.